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INTRODUCTION 
BY BARBARA CHAVEZ KUBAN , 
DEPUTY ASSISTANT SECRETARY FOR HUMAN RESOURCES 

Strara^'oc 1 ^ 3 "?, 1° weLcome y° u Co chi s Workshop on Evaluation 
resDonfiML^^ n InCervention Programs. Although my prime 

inJ i i ^ in man£ § emenC »nd in organization, as a 
parent and a grandparent and as a citizen, I am really interest- 

todaf TSS2 d ab °Vf qUaUty ° f ^ucation in oil cnoo! s 
today. I chink we all have a responsibility to find wavs to 
help improve educational delivery. ' 

Another concern of mine which I would like to share 

it i/?V S f° ne ChaC 1 SharS WiCh Secretary Hufstedler? I hope 
it is one of yours or will be one of yours in the future! It is 

cellel a^Ud" ty? t ° f f cho °l. Primarily an urban school anS s 
children of ore^L^^ ' ^ a 5 e ^oximately 7.2 million 
that rho ?5 a P 5 ^ i agS re ^ lrin g day care. To me this means 
that the trend will be toward seeing child care as a resonn? 
bility of an extended family, that family be?ng ?he emp^yer"' 
the community, political and educational institutions . 

active roiI h ? e ^ uca f ional community, I feel, must take a very 
active role in developing programs for preschool children as 

"hese tlTsT.ll t^T 0 V° W inCOme =«*"es. I believe'that 

system in thflnr t ° UV S °° ial SySCem ' our educational 

s>stem m the future, and are not prepared for that even before 
they gee into the ABCs . At the Department of Education? 1 
be attempting to establish a day care center to deal with 
special educational concerns and needs of this very young and 
important population. y • yuuu * dna 

I mention this to you because as expert:, and inter- 
would iu ^ucational field, I would hope that you 

would include these urban schools in your future discussions 
tor improving educational delivery. I also hope that I can tao 
some of you as resources as we move on ahead with this very iS? 
portant project. v^ L/ 



. I would like now to turn this over to Floretta 
McKenzie, who is the Deputy Assistant Secretarv for School 
Improvement. I want to thank her publicly f or' cont inuing to 
bring us together on these very important issues. If I Ly I 
also would like to thank Grace Watson of my staff at the Horace 
Mann Learning Center who is committed to bringing us together on 

cessfu. and we can share this information with a broader commun- 

FEDERAL ROLE IN INTERVENTION PROGRAMS 
AND RESEARCH AND DEVELOPMENT 
BY FLORETTA McKENZIE 
DEPUTY ASSISTANT SECRETARY FOR SCHOOL IMPROVEMENT 
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I Chink Che applause for Barbara is well deserved 
not just because Barbara's budgec helps ys provide such f ora , 
but because she is definiCely commicced Co che programs of che 
Department and noc Co jusc seeing that people are where they 
belong in chairs and Cables. ThaC is whaC makes a Department 
move, Chose of us who work CogeCher for common goals. So, I 
give a very sincere chanks Co you, Barbara, because chere is no 
other way for us Co P ull CogeCher such a wonderful program and 
such very imporCanc individuals Co work wich us on a viry cr^ti- 
cal copic. y 

We have had a series of programs or workshops on pro- 
grams co improve urban schools. Buc Chen we choughc that we puC 
programs in place and somecimes people expecc immediaCe results. 

Buc Che programs abouC which we are Calking somecimes do noC 
give us an immediate reCurn. These are programs chaC are direcC- 
ed Coward che prevencion of wasce, wasce of che mosc valuable 
resource of chis counCry— human beings. These people, more 
ocen chan noC, live in our urban communicies. Thev are ofcen 
poor. They are ofcen minoricies. We often have low expecta- 
tions as Co whaC Chey can do. BuC as an educaCor aC Che local 
and scace level, and as you who are in educaCion, we know chaC 
our charge is Co help every person develop Co his or her fullesc 
poCenCial . 



These programs are developed noC only aC Che federal 
level buc local educaCion agencies have inCervencion programs 
some chaC we don'C even know abouC. ScaCes fund inCervencion ' 
programs abouC which we do noC even know. If we were Co puC 
CogeCher an rnvenCory of programs ChaC are crving Co meet the 
needs or members of our socieCy who ofcen are' alienaCed , I chink 
we would have a very large volume. 

BuC 1 see as a pare of che role of che Federal govern- 
ment noc only che responsibilicy Co provide some of che seed 
money or venCure capital, if you will, Co provide opporCuniCies 
tor us Co bring new parCicipanCs inCo the school scene, that is 
business in a different way, to bring in parents in a different' 
way, and also to enable school people at the local and state 
level to use new strategies, new methodologies to try to meet 
the needs of a large segment of our population. 

There are three programs within the Office of Schoo 1 
Improvement that are very visible urban intervention programs: 
oncu7cir^° re blueprint; the Cities and Schools Program; and 
fLbH/tXLEL. These programs are evaluated periodically, and it 
seems Co me somecimes as if we joyously awaic informaCion abouC 
Che failures, or whaC it is that the programs did not do. We do 
have people here who are in the business of helping us look at 
these programs. In talking to the people who are in the busi- 
ness of helping us look at the programs, I think thev are lust 
as committed that we find out those things that are working 
well. But unfortunately others only highlight those things that 
are not working well. 
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t-hinoc . ? S was workin § in a school syscem and so many 

ar?Lr W M 0 801 ?? Wr0ng ' J haVe a Pawner here who was with me 
ac chac Cime. Ic goc Co Che poinc where we were Crying new 

^"SLSiT * onth }y' When know chac you are not meeting 
So ?hl r h a populaCion, you do have to try different thing!, 
nniv 5 federal Government must be into risk taking. We are not 

J~ y <2 ?i ln £ " lCh Ch °l e ChingS abouC which we know a L ° c > but Se 
iJE rh* ? business of expanding the body of knowledge, develop- 
ing the state of the art. That is what you are abouC if you 
1t«r l 3£° P ^° gramS ^at deal „ich improving urban educaCion. We 
just do not have all the answers, and we must not be frightened 

^!n c ng £ hS answers because the research or Chi evalua- 
tion cells us ac this poinc chac maybe chis is noC working. We 

ShaC°i, a rhp yS a i S ° n S Concerned ab °uC how many are we impacting, 

scare P n f C ; ° nCe y° U u C ° me C ° Che so ^t:ions, Chen you can 
scare weeding ouC some of che cosC facCors . 

Drover* n^rtl B l ede * al Governmenc are inCo small inCervenCion 
projeccs on che urban level and we are inCo some large ones. 

JfJ-SiS 88 /? d ° in TiCLe Ij Che Chings we do in ESEA , all are " 

|t improving the quality of education and thus the 
quaiicy ot lite for many people in our urban cenCers . 

FEDERAL ROLE IN INTERVENTION PROGRAMS 
AND RESEARCH AND DEVELOPMENT 
BY MICHAEL TIMPANE, 
DIRECTOR, NATIONAL INSTITUTE OF EDUCATION 

i-oii .,e h ° h : m 2 G ° d ' you mlght: sav ' researchers are going Co 
mSii t^° W develop programs again. ThaC was Cried once, or 

the l960?2.i? h S ° me K eeUng Ch ! C SOme ° f Che disappoinCmenCs of 
the 1960s mighc have been caused by a little over-arching ambi- 

^h«v ? ov er-confidence on the part of researchers who thought 
they knew exactly vhat we should do. S 

I guess one of the things I want to say today is that 
we have not been inactive during these vears . Believe it or 
not even researchers can learn something from experience abouC 

I thini ^ meth ° dS 35 WeU 35 ab ° UC Che ^velopmenc of programs, 
r. H h 1! 15 a 2 lm P orCant: underlying message in what Norm 
Cold and others of my staff who are here today will be saying. 
We have come a long way in 15 years in terms of understanding 
IIL L ?S I and /nterpret these complicated phenomena which 
make up the urba. education scene. 

pvn(lrf . a W %S l !° haVS 3 new °PP° rC "nity to demonstrate that new 
S off^L I 15 new opportunity is the Education Department and 
the Office ot EducaCional Research and ImprovemenC in which we 
sit. .ye have thought from che ouCset chac OERI offered an od- 

accessibL rl h° Che . fruiCs of our research more easily 

accessible through che organizaCior. of che federal education ' 
J V i£ les Co ch °se who were developing the nost risky but per- 
haps the most exciting new programs. We felt we would be in 
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closer relationship with people like FloreCCa McKenzU and with 
the others in the programs of the Office of School Improvement 
and, indeed, the other demonstration programs of the Department. 

ihat, indeed, is turning out to be the case. It pleases us 
greatly, because we think we have information to contribute, and 
the distance across which that communication needs to take place 
has shrunk greatly. This is very good. 

Research, or at least the research NIE does, has a 
couple of things to contribute to the kinds of programs whose 
assessments we are going to look at this morning. The first is 
that year in and year out at the Institute we are conducting 
research based on trying to improve the state of the art and the 
delivery of education services themselves. So, we are doing 
fundamental research on how kids read and how they learn math. 
We are doing a great deal of research on how tests are used. We 
are doing lots of new research on the acquisition of language. 
We have spent a great deal of time over the past several years 
trying to figure out exactly how schools as organizations work. 
These are alt more general themes of research which play in over 
the years to our understanding of how the particular kinds of 
projects about which we are going to be talking today can work 
or might work. 

The second brand of activity that we carry out is 
evaluation itself, evaluation of ongoing project's^* Norm Gold 
will be speaking to you in some detail about the methods we have 
developed to conduct such evaluations, because we aje very sensi- 
tive to the particular needs of innovative urDan education pro- 
jects to an evaluation which has, to start with, carefully 
identified the objectives of the program and just what would or 
would not constitute criteria for success. No research or evalu- 
ation effort can come charging in to the dynamic and living 
scene of an urban intervention and try to figure out exactly 
what's going on without long and careful discussion and under- 
standing of what it is that is intended in that intervention. 
So we have developed a fair handful of evaluative efforts of 
this sort. 

About five years ago, we began to evaluate the career 
intern program of the OIC, which, at that time had been set up 
in Philadelphia. Howard Resnik is here today to tell you that 
we have since taken that intervention to fou*) other sites at trhe 
request of the Labor Department and are jkJw jdoing one of those 
great rarities in the history of research, /n honest to goodness 
replication and evaluation. We have ^fctken al^success ful interven- 
tion in one site and have ins talle-er it in four others, and evalu- 
ated it to see whether or not itrs success is attributed to the 
site in which it first succeeded or to the program itself. 

We have been evaluating two of the exciting programs 
that Ms. McKenzie mentioned, PUSH/EXCEL and the Cities and 
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Proer flm , K. devel °P ed Che Experience-Based Career Education 
Program a few years ago, about which n-.any of you know, Co trv to 

espe e cla P nv"on°the S TOCa «° nal Gaining, k concentrate" 
especially on Che urban scene. 

Most lately, we have participated a good deal in the 
IhT^fVp y° uch . ini t:iative. As you know, it ha passed 

caLnS^r whfn f reS ? nCaClV ! S and WiU be P endin § on ^e Senate's 
calendar when it returns after election day. This is an at- 

aTthL ^onnf 1 "' £° ? SVel0p 3 nSW intervention effort, aimed 
SLSS Seco " dar y ^hools in our cities. This is a desperately 
" « focu ^ i c feems to me. I chink we have been able to 

2?S 3 g0 °u d f aL C ° Che desi § n of chaC program in a way 

which stresses the literacy and employability skills that it 
seems to us the kids in those schools need more than anything 
else. It stresses the need for the whole school to be involved 
Proeiam 8 orf ^ ^1°^™* f unda ^" C a 1 Ly , not in adding on another 
progiam off in the corner somewhere. We hope to begin to impart 
those skills to those young people. That is a very 8 excUing 
prospective intervention that we hope to continue to be involved 

In the last few months we have begun to forge a 
partnership which excites us very much with the FolloS Through 

effort. In H hlC ? We be abl£ C ° c °nt:ribute to that program's 

efforts to develop new or modified models of intervention for 

l^L P f° 8 r: am % f0r ChS P rimar y ^ades, which Follow Through has 

iom? If rZ Ve t 10 ° Ur Ci ^ iSS ' S ° ChaC we ma y a * ain contribute 
some of the basic insights we have gained on effective instruc- 

schSo s S t o aCe8 ^rH and effecCive organizational strategies in our 
schools to maKe these interventions sail. 

fence f i f ^"J herS C ° day ?! bring Co y ou > we ho P e • exper- 

ience from half a dozen major effor. to see how the tools of 

f!J!£?S 1-? evaluation can bear upon the success of these urban 
learnTd S P ro § rams ; We wanC to tell you that what we have 
learned has been learned continuously over the past 15 years, 

lSc ,M, haVe P , t0 Say toda y> as 1 am sure Eleanor Farrar's 
remarks will reflect, is a different story than formerly was 

APPROACH TO EVOLUTIONARY PROGRAM DEVELOPMENT 

BY ELEANOR FARRAR, 
VICE-PRESIDENT AND SENIOR RESEARCH ASSOCIATE 
THE HURON INSTITUTE. 

Evaluation used to be a modest cottage industrv in 
American education, but now it has become a big business'. Twenty 
years ago, a meeting on urban school reform would have been con- 
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cerned with curriculum revision or Graining ceachers and ten 
years ago, a meeting on school reform would have been concerned 
"i^h "forming the organisational climate or with improving in- 
terpersonal relationships among students and teachers. 

*™ u BUC 5° day We are here Co Calk abouC evaluation. School 

I' e „ rS / n >f raCtitionerS aLike seem Co be aL ™st exclusive- 
ly concerned with program evaluation and assessment, and issues 
that involve testing. Program development seems to have gone 
?hf„S •f Cyl f' P arClcu larly directed development efforts. But I 
think it makes sense to pause for a moment to think about just 

ZSSlv iS ? C WS evalua,:in g> "hat it is that we propose to 

apply our evaluation methods to. 

e ., a , The school reforms of our time are far more comprehen- 
sive and complicated than they were a decade or two ago. The 
relatively straight-forward curriculum reforms of the earlv 
Hiri l ™°J vin 8just stents and teachers- curriculum reforms 
! h ' J c turned out when Bob Stake took a closer look at them, 
were not so straight- forward after all— seem simple in their 
conception and design by comparison. The new generation of 
school reforms involves not only students and teachers, but 
parents, members of the community, occasional social service 
c?K« C t eS '/ increasingly employers as well. The new wave of 
It -Sii-ES programs is a good example. Cities and Schools 

is another example, as is PUSH/ EXCEL and Experience-Based Career 
Education. These programs tend to blur the boundaries between 
schooling and community. They are enormously complex social- 
interventions and they involve many actors. 

So, at the same time that the state of the art of 
evaluation has become more complex and the methodologies of 
evalua ion have become controversial in their application, the 
programs have become more complex. So have our understandings 
?m«V programs and the way that they are developed and 

implemented. Nearly gone are the days when people think about 
till IJ P-!1 S Slmp y a chre e-staged process, in which you pre- 
tltl* ^1 . ' y ° U appLy Che CreaCm ent under some kind of condi- 
tions that are as nearly controlled as you possibly can get them 

?? c £p an ? th o n P° sc - Cesc c ° see if the intended treatment had 
its effect. By the same token, simple conceptions of program 
development have been swept away by much more elaborate formula- 
tions about how the process works. 

I'm going to talk about both the old and the more re- 
cent ideas about program development and implementation and our 
notions about how this process occurs because I think it has 
important implications for program evaluation for what purposes 
these evaluations realistically can serve in providing informa- 
tion about progran progress or success. 
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In brief, I would like Co syggese^that Droerams are 

tion and Chat, for all practical purposes, there are really few 

L ° ns beCween Program development: and implementation. 
Programs developed in R&D laboratories or in demons era cion 

S an 5 e scSo r oJs r d T T L ° P ^ nCe ChSy geC ° UC ^oTolTl dis- 
tricts and schools. The process is continuous. The more 

oc^r cha^rhe^ inhev fi* vi ? w > is Chat: one stage follows Che 

u . program Is developed in a laboratory, ic *oes 

through some kind of development: sCage and Chen is impleaded 

schoo ll "rVT, 0 ? 1SSS high fideLic y fashion in local ' 
S : n Buc whac I am proposing is chaC chere is no fixed 

implemented by schools; raCher, che schools develop 
variations on a sec of policy ideas or improvisaCions on a oro- 

£2^3 V WhiCh iS P rovided °y ^e Federal Government or P 
provided by program developers Chey supporCed. 

I call ic a cheme because ac base it is a sec of 

than" Lv^KL^ci"' 6 ^ 10 " 3 ab ° UC im Proving schools, incencions 
£«5p fS2 j f a P ed inCo operacing programs. The Chemes may 
have form and structure and set procedures, that is, they may 
have a blueprint, such as the ExperienceBased Career Education 
model or che Career InCern model, or it may be jusc a loose sec 

with tL 3 ?" h °Y-° im f r ° Ve educaCion > something ChaC comes 
! ?h! i K 16S an ? regulations to steer local development, such 
cLeiv r?* M CC * J C m i ght be V6ry s P ec ific or ic might be'reia- 
tively flexible and unformed. But, in either case, local imole- 

001 cv'intfir? 5 3 " differenc ^ape to the progrlm or ? 

policy intentions and the program evolves. 

rp „ . r . Development of implementation is an evolutionary pro- 
cess with a beginning, but a very uncertain end. But if pro- 
grams are thought of as evolutionary, how can they be evaluated' 
Ar L a Jr e the , cr ^eria one uses to determine success or failure? 
purposes? Clm6 d6Cide C ° evaluaCe and for what 

The answers to these questions seemed relatively 
scraighc-forward when people held a simpler view of program im- 

e'fro^s^ino IS in£ Z™ ed mOSC ^™ development 

efforts during che 1960s and righc chrough mosC of che past 

p!h!£i " aS tUou Z ht w °rk in Che following manner. The 

bl?!2i f° ve rn m enc--usually ic was Che Federal government-esta- 
blished . sec of policy ideas for reforming American schools. 
These were usually a bunch of ideas ChaC had been floating 
around in che social science liceraCure aC che Cime, and they 
were ideas picked up by Che Federal governmenc. The Federal 
governmenc funded developers Co turn these ideas into a bunch of 
operating programs that could be -\mplemented locallv. Thes° 
or^HLon^ d ? vel °P ed in research and development' laboratories 
or in demonstration schools or lighthouse districts and they 
resulted in a set of plans or blueprints, and they were accom- 
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panied by curriculum guides _na instructional materials and de- 
tailed procedures for local practice. 

Sometimes the programs were accompanied by training, 
sometimes by technical assistance from the developers and some- 
times they were not accompanied by these things. But in most 
instances the programs had undergone extensive eval unions and 
assessments to determine that indeed they worked, and indAed 
they were ready for broad-scale implementation. The programs 
were certified. 

Once the programs were determined to be ready, it was 
expected that schools and school districts would implement it 
pretty much as was planned, using the materials and procedures 
and embracing the same goals for evaluation. The ideal was a 
high fidelity replication of the program. Local schools might 
make a few modifications and changes in the program or in the 
blueprint? to accommodate local circumstances^W they would not 
tamper with the innovation in major wayw^tVwas proven to be 
ertective only if it were implemented ai- £he developers had in- 
tended. / r 

This was a handy way of thinking about program develop- 
ment. In particular, the notion that programs were more or less 
fully developed before they reached lofcal schools was consistent 
with the methods of systematic evaluation studies. This view o ? 
programs was sympathetic to the methods of science. There were" 
formal blueprints and procedures' to use\ in assessing the effec- 
tiveness of implementation. The programs had specific goals, 
often linked to outcome measures, and the treatment was directed 
toward the goals. Students were pre-tested and post-tested and 
the lack of gains presumed that there had been some kind of 
breakdown in implementation. People were not doing what they 
were supposed to do. Failure to achieve the desired results 
was caused by failure to carry out the program as intended. 

Arguments sometimes ensued about the appropriate 
design for the evaluation or the appropriate goals for the pro- 
gram. But one thing seemed clear, and that was the program, 
as it was developed, represented the best way to implement 
federal policy in a way that had been certified by program evalu- 
ations. Thus the program design was completed once it was out 
to local schools and in that form it was stable. If it were 
stable, it could be evaluated using fairly rigorous method*. 

These ideas about program evaluation began to gi,e wav 
in the 1970s at a time when evaluations of most studies began to 
suggest that they were not succeeding and when implementation 
studies revealed that local districts were making wholesale rr.odi- 
rications and changes in programs. Ideas began to shift at a 
time when increasing attention was being given to the role of 
local actors in implementation. Research on behavior in organiza- 
tions and several federal studies on implementation suggested 
that it was more sensible and more realistic to pay attention to 

ERIC 10 
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Che fact chac local people have particular needs and preferences 

ences^Vh" C ° WhiCh ^ pr ° gram musC be accommodated pre fe" 
n2 C ff ° h wer ^ noC alwa y s c onsiscenc „ich chose of che 
developers or chose of che Federal GovernmenC. 

, pxrq T * n addici °n> people began Co Calk abouC local con- 
texts. The program was being incroduced Co a very parcicular 
local concexc chac varied from disCricC Co district Ind U onlv 

Toca e i S c e oncex^ a if^he re ™ l * ^ C ° b , e Ch ^ eS S -co^a'te" o 
Furthermore %if f~ P ro § ram ^as Co thrive, never mind survive, 
furcnermore, che local context consCanCly was changing as a re- 

enc^S e ~ act P o liCiC ? L ' and eC ° n °^ C changes -d e X e " 
lence had an impacc on implementation as well. As local dpodIp 

working out. Components were massively revised or somecimes 
abandoned altogether; curriculum materials sometimes were lied 
were used by some staff members and not by others? or were re-' 
vised to meet the needs of local students! InTffecC local 
staff redeveloped the program to meet local circumstances Some 
times these redevelopment efforts were well planned and care- 
fully planned; but, more often, staff stumbled upon or invented 
new ways of doing someching and chey did Chose things in a some- 

develoomenr PiC ?H S Wy ' In ^ w ^ s ' Chou S h > thosl were 
developments. The program was continuing to be developed, and 
as the programs continued, they were implemented. 

It seems a more accurate description of reality to 
«mf program development and implementation as part of the 
same process, rather than two separate stages. It is an evolu- 

best a gr P the C i? S o a rr Pr °? ram SCa ? f Ch °° Se C ° do whaC ^ ChlSk is 
best for their particular setting. As they make decisions about 

evol?e r s 8ram 33 Ch6y 3CC ° n th ° Se decis ^> the program 

This way of thinking about program development raises 
tion.r? q T Ci0 ?/- 0ne . is che following 5 If programs are e'otu- 
Cionary and implementation inevitably leads to change, what does 
this^mean for evaluation? What criteria do we use in iudSnT 
success or failure? The program's survival is one criterion?^- 
lhac was one chac biological evoluC ionises used. Buc ic does 
not cell us enough. Survival is imporCant, buc survival in whac 
torm -- an improCanC quescion, parcicularly when Che programs 
are being supporced wich federal money year afcer year. 

Anocher quescion Chac this view of program developmenc 

SSn^v^TH Ch f inv r Cme " C 0f LargP amoun ^ s ° f ^deral Lev 
into directed development efforCs. If thousands or millions of' 
doLLars are spenc on curriculum developmenc or program R&D Co 
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produce a product for Local schooLs, is it a wise investment, 
given the wholesale revision and modification of the products 
that ensue? r 

I'll tackle the second question first because I think 
it is one of the more important questions for education policy 
R&D these days. My answer is yes, but investment onlv up to a 
point. The directed development of programs for school improve- 
ment is important because the programs provide local staff with 
a program prototype, with a conception of the program that they 
use as a central frame of reference for their own implementation 
activities. It provides guidance. It's a blueprint to be tried 
in parts in varying degrees or in its entirety. It enables 
start-up to occur more quickly and by people who do not necessar 
lly have a clear idea of what form the innovation might take. 
Many of the people who are invoLved in these programs are not 
educators and they are not familiar with education prbgrams . 

Education program prototypes introduce new ideas, new 
materials, and new methods to schools. Though perhaps onLy some 
are used, they encourage new organizational arrangements by de- 
scribing them. District staff may accept or reject the blue- 
print, but regardless, it stimulates a whole bunch of ideas 
about what will or what wilL not work in that particular set- 
ting. Developed programs provide local districts with a theme, 
a theme that will produce many improvisations. Sometimes the 
improvisions are more appealing than the theme. That certainly 
has been the case in musical or literary history. But the point 
is that these improvisations represent a creative interaction 
between a central idea and those of locaL school people striving 
to improve on the theme or to elaborate it to fit a set of locaL 
preferences, locaL style or way of going about things, or LocaL 
context. * 

' p he view of program development and implementation 
as evolution and improvisation increases the need of local staff 
planning time, staff training, and technical assistance. 
Planning time is particularly important with this new generation 
of programs that involve many diverse interest groups convened 
together to work on issues that are unfamiliar to many and in 
ways that are new and untried. They need to get together to 
work out new organizational arrangements, to plan sensible di- 
vision of- labor, to decide on areas of shared responsibility, 
how things are to be carried out, and so forth. It is a long 
list with which we all are familiar. The more complicated the 
programs or the initiatives and the more diverse the groups that 
they involve, the more planning time is required before local 
groups start to implement the program with students. 

Technical assistance and training also are important 
in the implementation of complex school intervention programs. 
Both ordinarily focus on helping with the installation of pre- 
specified programs. But broader conceptions of technical assis- 
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Sii^!^ re^uired^ 1 ° f between 

^ ?? Sira ? Ll l " anv kind of complex work, chosl things § 
Chat we all wish we had gotten and sometimes do on our own It 

c J'! arly , neCeSS3ry u When manv Participants are un?am liar 
with the new roles into which they have been cast and they have 
little experience with these programs or others like them It 
cL?^ C - CUlar ^ im P° rCanC in "eakly specified broad-scaTe 
ence or'^orint 3 ' -S^?" 6 l ? n ° Programmatic frame of refer- 
mentaclon! P ' * USUaL accoutrements to guide imple- 

schooU „Jl! e f e f rS LoCs .o f Sood reasons for providing local 
schools with implementation assistance during the early davs of 
J program s life cycle. But, suffice it to fay that the evolS- 

f 0 ra a sLstaL e °- P ?° 8ra ^i mplSment:at:i0n does no " obviate the need 
assistance of rh* anvChing ' " enhanc " the need, but not for 
assistance of the usual sort. People need not only instruction 

™uC h °Ll\l?o l lS^ elsehaYSIvIseo °" 

TheV win ?2n3! • P ln devisin § improvisations of their own. 
JSc Jii i m Pro vi se in any event; but in many cases, the more 
ideas they have, the richer the improvisation might be 

uhar . T !? is brings me back full circle to the question of 
what do we do about evaluation when we conceive of programs as 
tron UCl °? a ch y /M her Chan - 3S P—Pecified for uni! io?m°f mpTem^nta- 
serves as the frZl^T "f ^ ^ iS being implemented" what 
effectivpn!^9 tT ° f refer ence for making assessments about 
effectiveness? If program goals shift as local staff find that 

fJSS%£? S ° me ? hin P Chey can d0 becce r ^an others or as che y 
find that some local students need some things more than others 
what do evaluators look for? Pre-test data may become obsolete ' 

i Hells ?f a not Cha "PA T j me fram6S f ° r ^alLtions maymake 
IviJSZ • i mUCh has been accomplished, and all the while 

Ch * - " ' 

a PP roaches M che n L W ct r er%^ ™ 
SmS 1 "/" dlversiCv and ch ange, one cannot expect stJb" 
nr^M 3 °£ referenc ^ Old-fashioned evolutionists solved the 
problem by asserting the survival of the fittest or some other 
doctr.ne of progress but I propose no such thing. One possi- 

! J° n y partial, solution is to admit thatthere are 
-hit ILSl success and that no single criterion oan comprehend 
.hem. Another possible, but quite uncertain, solution is to trv 
to frame processual criteria for success, using such notions as 
problem solving to judge whether good things are happening But 
another is to admit that no singil intelligence can^mpShend 
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all the possible ways that programs can be successful'. Many 
different criteria representing interests of many people would 
tell diverse and divergent stories about program success. But 
if we can envision these and other solutions to the new problems 
of evaluating programs, it would be rash to pretend that we 
could pick the best one* 

DR. JAMES COMER, Yale University: I did not under- 
stand your last term, processual, I believe. What do you mean 
by that? 

DR. FARRAR: If one possibly could figure out some 

ways of assessing process, a process that we could use, as a way 
for carrying out some kind of program or making decisions about 

it and determine the process to be effective or not. 

DR COMER: And then looking at stages within the pro- 
cess? 

DR/ FARRAR: Well, I was not so much thinking of stage 
theory as much as I was thinking about. uhat are constructive and 
unconstructive ways to go about problem solving or decision 
making . 

RICHARD ZUSMAN: I'm concerned with your last state- 
ment about the process. If you have limited resources and you 
are trying to make a decision of where to put those resources, 
where to put those dollars, how do you make that decision if you 
just have process information about the implication of a program? 

DR. FARRAR: I think the decision would rest, as so 
many decisions often do, regardless of whether there is hard 
data or not, on what we might call clinical judgment. You have a 
notion about a process that seems to be working. I mean, God 
knows just exactly why it seems to be working, but you have a 
sense that there is a process, a healthy process or constructive 
process of decision-making that's going' on and you decide to 20 
with it . / e 

MR ZUSMAN: I guess I still feel uncomfortable about 
that. You're saying they have three healthy processes going on. 

DR. FARRAR: That's right. Hopefully there would be 
more. If you only could fund one, then I would be glad not to 
have your job. But you're quite right. There could be a whole 
bunch of them. 

DR. COMER: What are the criteria for health? It seems 
to ir.e Chat there has to be some stage consideration and that at 
some point you would determine that although it looked healthy, 
perhaps it wasn't and didn't get too far. 
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, lh . How would you determine whecher or not a program were 
healthy and what: criteria would you use, che process? 

rn P5r aM . D , Rl FARRAR: I chink what: I would do, first: of all, is 
Suri^ rLr ? reas ™ably long planning time, expecting chat: 
during that: planning time or pre-time, call it whatever you 
there mi^Th* hen people are trying to get themselves together, 
that 8 ^ ,h a 1 } 0C °f ^"actions, exchanges, or behaSlor 
that we would call nonfunctional if we saw it happening three or 
four years later. But at least in the early stages we let 
people mess around while getting to know each other and decide 

of what"! i C Chi/S P ° inC we wouLd have some notions 

wrtlcSur cVil r r C CLVe decision-making process is like for that 
M^fl ciimate or that particular context. I'm sure it 

p?ocess a In'Buf^ln Ure f^K ^ /° uld be 3 g0od decision-making 
S Lh-?? 10 W - Uld be dLfferenC from one in Detroit which 
would nor 11 r T? 10 !! urn « fr0m 0ne in Salc La ke City. It 
rn ht anyCnin S c hat was uniform. The criteria would seem 

rLuthr - h ! r W3S W e rking f0r l0cal P e0 P Le > whether they 
moving' something that was constructive and forward 

a, a . MARJORIE HOACHLANDER, Institute for Museum Services- 
As a researcher I must tell you that you have stimulated my 

" ^ 'VS 6 m ° rning ^ whaC 1 wiL1 caL1 a theoretical 
ttl LEw V- do agree with you about the historical change in 
It's a JPii" f°; ° f ? sCablish ed programs in various localities 
Its almost like a fish swimming this way, sayine hi to a fish 
swimming Chat way because the other fish goes into Michael 
Timpane's world of making HIE empirical evidence to a consider- 

cation e ?f ee „ C ° * ubsCanCiaCe not only the funding, but the repli- 
cation of more things in more places. 

f- ho o™ ThaC L J " here my Caffy P uL1 rescs ' I f you accept vour 
theory or your desire to do this in this much more flexible y 

diverse way, how then do you maintain some evidence of emoirical 
documentation? How do you get to that point where somebody 
else who has not tried anything before, begins to think in 
^! m s?^ em P Lr !: C ^- eVLdenCe because Chat is the way some of us 
rnr, r„ ?u a l e Z tQ i ^ ° Cher areas of our Chinking? How do you 
core to that kind of mature compromise and design? 

hal . . FA JRAR: I think as a starter the more people who 

talk about it, the more it will help, people who are solely 
^ V H^f° n em P Lrical evidence, to perhaps pay attention to other 
kinds of evidence. But I think that it's also useful in evalua- 
ting a program to do more than look at empirical evidence, and 
by that I mean quantitative evidence. 

MS. HOACHLANDER: I'm not being solelv dependent on 
cooks 1 m Saying h0w do you combin e the two into a recipe that 



Page 14 

MR. FARRAR: I chink chac different methods and dif- 
ferent points of view can be brought to bear on the study of any 
single program. If you take a single -program-,- -like Cities and 
cally ° C ° evaluate ic quantitatively, empiri- 

MS. HOACHLANDER: That's not synonymous. 

MR. FARRAR: No, no. Quantitatively. One can apply 
qualitative methods to it. People who do both of those things 
can be evaluation researchers, social scientists. Evaluation! 
also could be done perhaps by social scientists from the perspec- 
tive^: °? er P e °P le who are involved in the program. They 
could be done from the perspective of teachers who are working 

c« fi^£ r ° 8 S m ' k u ds Wh0 are in che Program, parents, so on -and 
so rortn. Then what you have is an array of findings. 

Now, that does not lend itself to a neat bottom line. 
It somebody wants to know is it succeeding or is it not, then I 
think the answer would be well, if you talk to so and so, it is 
this, and if you talk to such and such, it is that. It is 
messy, there is no doubt about it. But these programs are messy. 

e a ^ ^ MS i McKENZIE: The y su-e are not neat for us at the 
tederal level, and I see the members of the national PUSH staff 
here, and others, and they know that they are not neat either 
since they are working with so many different communities. 

EFFECTIVE METHODS OF EVALUATING URBAN INTERVENTION PROGRAMS 

BY ASA HILLIARD, 
CALLAWAY PROFESSOR OF URBAN EDUCATION, 
DEPARTMENT OF EDUCATIONAL FOUNDATIONS, GEORGIA STATE UNIVERSITY. 

In chis shorC period of time that I have, I have proba- 
bly chosen to cover too many things, which means that in some 
places I will be necessarily broad and in others maybe overly 
specific. But I chose to err on the side of trying to do too 
much rather than too little, so please forgive me if I am not as 
complete with anecdotes as I could be. 

Our educational system has yet to become distinguished 
in the sense that achievement level for most public* school 
students can equal that of their counterparts in several other 
nations. As a nation, we have not yet been able to develop a 
mass public school education program of the highest quality. 
Where we do have high quality, we have it for the few and for 
the privileged. 

u nJ A Phi DeLca Kappa article in the current issue reports 
that 96 percent of the students in the Soviet Union complete 
high school, a ten year program, rather than twelve, which in- 
cludes calculus and a foreign language for the vast majority. 
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AC Che same time, the article reports that our own schools oro- 
duce graduates of only 75 percent of the students? Move inures- 
S! nlJ re P orC u Of a ^ s ^vey which revealed that over half of 

d " S S T ch00ls f>™* rSC > uire one y**r of mathematics f or 

ftfon ?2; f Pan and German y also outperform us in public educ 
?, 0 0 ?' J eC > 1C 13 our nation more than any other which spends 
rreely of its resources for schools of every conceivable type. 

tinpni<=h 0 H If ° Ur P ublic . sch °° L syscem as a whole, has been undis- 
tinguished our evaluation of these schools formally has been 
even less distinguished. Little has been offered to educators 
from evaluators which can be shown to have improved pedagogy 

Nation hL b K 6n n ° dearCh ° f lite rature in evaluation. SfSher 
nation has been so preoccupied as ours in this area. And yet 

for* y £i n§ ' ?u haVS creaCed mysteries where none existed be-' 
tore. Then, the more we explore the mysteries' of pedagogy the 
deeper the mysteries have become. peaagogy, cne 

■ Few urban school systems, if any, are thought of with 

conLcted'on Che . bulk ° f educational research has been 
conducted on populations in urban areas. Researchers have 
Sffli • magnitude of failure in urban education, but have 

St on datf?r ra i C ° de " ribe iCS dynamics ' If we appeal Co eval - 
Tal £or r f- 1 ? S edu " Cors ' ^ tends to be as source mater- 

ULl "penalization, for curiosity, sometimes for punitive 
leverage, but certainly seldom for pedagogical guidance 

?rpar Hoa i L ^ k I a pp Che P°P ular u studies. Schools do not make a 
n dea p i difference in children's achievement - Coleman, 
Jensen. Headstart programs can't change- things for poverty 
children since poverty children can't be changed — Jensen and 

Si ih£! ?H Se *, t S ° ne l00kS aC 0lder l^erature in p i 
one should not be surprised that the scholars reflected the 
socio-political thought of their eras and developed expectations 
i lk ?H S K me ° f Che ceache " of low income children? Soweve? 
wTre^ked so late^^ ^ and P- si -nious questions' 

citv nf pi r^ nn °5 l ? elp buC u 6 im P ressed a C Che profound simpli- 
city of Ron Edmonds' approach to the study of urban schools -- 
simply look at the ones which do well and find out why. Similar- 
b?Av?h!h Be §^^ng Teacher Evaluation Study and others like it 
provided a model for inquiry. These simply have not been ap- 
plied fully to urban school research. The work of Ray Rist, 
Rosenthal and Jacobson and many others have shown empirically 
that lite in classrooms is anything but standard for children. 

Why did so many evaluators ignore, overlook or miss 
this important point? The point still is being missed in the 
main, inspite of such important work as cited above. I believe 
there are many things about the way the work of evalaution is 
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conducted that make for major impediments to the work. Yet a 
parsimonious look at these things should illuminate basic 
problems all at once. To take this tack would' be to guarantee 
the improvement of the evaluation enterprise overnight, and 
therefore would improve urban education evaluation as well I 
hope that I can describe a few of these things sufficiently for 
us to see them in somewhat more bold relief. 

Number one is street language in academic evaluation, 
tyery first year graduate student knows that terms must be de- 
fined if research is to proceed. Presumably, evaluators who 
sometimes teach them know this, too. Yet, while the evaluation 
designs may be constructed with unbelievable rigor at certain 
points other key things in the design are frequently treated 
..very, .shabbily. My mother always told me that when you took a - 

iV an 5u PUC ° n Clean cl0Ches » y° u h ad to use clean underwear as 
well. The same principle ought to apply in evaluation. For 
example, what does school success mean in an evaluation design 7 
What is a program? What is race? These terms seldom have either 
precision or common definition from one evaluation Droiect to 
another. v J 

The consequences of this, however, are tremendous. 
Let s take the word "program" for example. Headstart is a pro- 
s £ am ; Teacher Corps is a program. Follow-Through is a program. 

Title I is a program. But really, these are simply names of 
categories of fiscal support. They cannot be described uniquely 
in operational pedagogical terms for some of the reasons that we 
heard earilier because a variety of things may be done, not 
necessarily bad, under any one of those headings. So there is 
no real problem with that unless evaluators want to say some- 
thing valid about "the program." 

Most research on urban education, by its failure to 
articulate the unique pedagogy of urban intervention, allows for 
the development of erroneous, implicit assumptions that urban 
intervention programs offer unique or better treatment than a ome 
norm or group with which it may be compared. This was a problem 
for example, with Jensen's research on the boosting of IQs.' At 
no point in his analysis did he deal empirically with the nature 
of the treatment which children whose IQs were not boosted had 
received. Rather, he looked only at child factors for an explan- 
ation of his findings. He does the same thing in his new book, 
Test Bias . 

Evaluators pro and con in urban education program eval- 
uation have been consistently sloppy in overlooking or in 
failing to describe the real "treatment" which the subjects actu- 
ally get. When evaluators use the language of the street and 
allow a single street label to represent a wide variety of opera- 
tional definitions, then reliability sufficient for scientific 
measurement cannot be obtained. 
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Porter ^ f? conCenC - v alid testing instruments. Andy 

world w?S ^ collea §^s should have startled the evaluation 
world with the presentation of his very rare emoirical data nn 

^rteTanTh^o^ ° f sC ^ardized- te'sts of Si™? * °" 
porter and his colleagues were able to demonstrate that there 

'tonics ?LV n 3Verage °5 3 50 P ercenC mismatch between the 
k™w 5 C WS f e covered in popular fourth grade arithmetic test 
k ,J? P°? ular standardized tests that wire used to Ivaluatf 
the children's progress. One wonders if it is this bad in thl 
well-defined and somewhat sequential arithmetic content area 
what we would find in the language arts or socials tuSies I?eas . 

cr , r BuC 1 wonder even more why there are so few em D irical 

rinding ?? ntent validi ^ against which to compare the^orter 
L "f S ; rr ^ seemS 1 C0 °.P aCenCL y obvious that th content'vIlU 
dity ot program evaluation instruments must be es ablished not 

^ the faiW C to PC d e o tHi £aith \° r £? sed ° n e *P- C °P^°" S 
-s cne railure to do this simple thing that has resulted in f-h« 

PrLnLT? teSCS aS o the Peabod y PicC ^e Vocabulary test and 'he 
lid ^h° L r " vent .? r y ? c °^s as criteria measures for Heads?art 
dete?miL r tion C ?h C '? nal inCe ™entions where' there was no empir cal 
rhrZl i °" ChaC any P ro S ram "nder the Headstart label had 
those criteria as program treatment objectives. 

The same may. be said of the use of SAT scores for * * 
sessmg "school success." No matter how laudab e thl Lal of 

choof curricurwiu" ^d^ ^ b& ' Che content EsSSJh^lth 
be° 

::aJuation e e1^rt eaSUreS ™ ^~ble pr^quisitVtT ' 
Third is testing for status or for chanee Aoain' 

operace, we do not act as if we know. Urban intervention nro- 
grams reqmre tests with 'content validity for the intervention 

a^onf ^iL^of"^" 8 "^ se ? sl "« " Provide discriMna on' 
among levels of achievement along all parts of a total ratio. 
Moreover, the sensitivity must bl sufficient to detect chants 
III tH P / r£ °, rman T ° V f r Sh0rC and lon § P eriods of time. Affer 

Next is outcomes or process. Urban intervention D ro- 
gram evaluations frequently are little more than a formative 

il^fvr Promised process is in operation T^ere 
is little accountability to such a scheme, either for the oro- 
gram or for its evaluator. The effectiveness of urban interven- 

v musC focus on learner outcomes in the final 

analysis with process evaluation as a part of that analysis. 
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^ A friend of mine who was funded for several years to 

conduct an innovative teacher education program for urban areas 
had abundant data on how well the trainees in the program master- 
ed the program competencies, but no data at all ever were 
collected to determine if the mastery of those competencies by 
the teachers and the application of the competencies with their 
students resulted changed student outcomes. This person never 
attempted to initiate such data collection nor did <any donor 
ever require such. 

Next is levels of aggregation. Frequently, the things 
which could be learned from even the meager data which are col- 
lected on urban intervention programs are collected in such a 
way ks to be totally confounded. 

For example, it is important to associate the learner 
outcome data with the specific treatment which is or which could 
be responsible for the learner outcomes. Yet, data on student 
achievement are collected frequently and aggregated at least one 
level beyond that where the treatment actually occurred. So 
learners experience classroom activities or small group activi- 
ties within a o classroom within one classroom for the most part. 
Every learner experiences it within one classroom not in the 
district. They do- not experience all the classrooms in the 
schodl. Yet the achievement data may be aggregated at the 
school or even at the district level, obscuring essential in- 
formation . 

In fact, it is this very practice of reporting average 
performances by school or by district which was contributed to 
the widespread belief that children who live in urban areas or 
in culturally homogeneous neighborhoods are uneducable. Disag- 
gregating the data would reveal in many, many cases that the 
story is not uniform for all classrooms. Many teachers have no 
trouble at all teaching the n unteachables , " Many schools as a 
whole also have performance records which may be lost in dis- 
trict-wid.e aggregation, as Ron Edmonds 1 work has shown. 

At a national level, it also is true that performance 
may vary widely from one project site to the next, I can give 
another example of a different type. A national agency has col- 
lected statistics on the proportion of black children who are 
assigned to classes for the mentally retarded. They also have 
figures for males and for females in their breakdowns, Yet, 
they did not set up their data system to yield the most single 
important fact of all, which any clinical inspection of EMR 
classes would suggest. This fact is how many or what proportion 
of the children in EMR classes are black males. The particular 
aggregation w/hich was selected obscured the main event. For 
example, the radical disproportion of black males in EMR classes 
undercuts the Jensens thesis of the racial basis of IQ test 
score"* perf ormance , or at least for EMR placement. 
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The problem here and wich other chines obscured 
.through aggregation, is frequently there is a type of Political 
done' SeL"? ' ch f ^~ appropriate agg-gatio'n ?roS°being 
amSne cUssrniL°fh ^ fear u ChaC che deferential performance 

on Che n ^r^^ SingLed ° UC for revi * w > sometimes 

on cne assumption Chac Che bad material, meanine che children 

is not randomly distributed, and therefore a gold teacher mlv' 

maceriai "V!? '2 l00k beCaU5e or'she'hL^d 

disoron^r^i ^pp 336 SUCh as the case ' of che black male 

?ir?«2 P ? ' he differences are so blacanc chac che inCerpre- 
for eVllLllZt eV ?J uaCors and, others inco sensitive CerricSry 

me^C ?eierL TK 8gregaCi ° n LeVeL musC maCch che c reat- 

menc level ac lease. Then you can do whacever else you wane. 

NexC I will discuss mychical sequences and mvthirfll 
componencs We must cake note of che widespread tendency Co 
cninLoH S f UdenC learnin § as following sequences or as being 
composed of componencs which do noc come from empirical observa- 
tions of learners. I'm really happy Co hear what Mike was «v 
ing about the direction of NlLreseLch, buc c£ac has no? ye T 

aoplar cS b e LC L eVaLUaCi S n - T 5 ese s ^-nces and componencf 
appear Co be more a produce of Che logical exCrapolacion from 

cions. When evaluacion accivicy is expecced Co give d?Ca for 
diagnosis, we muse quesCion the validity of certfin popularly 
accepced sequences and componencs. "ream popularly 

noc onlv a F ?p^ X ^ Ple ' "^ st:s . of reading" frequency include 
ing skills Lu com P rehens ion, ^t: a cesc of so-called read- 
Id rn hi The readin 8 skills are presumed, not observ- 

Such skilirirword 1 ^,' 0 ^ 116 f ilit:y C ° reaS 0r Co comprehend 
iucn skills as word accack or deCecCion of beginning and endinp 

parScuS S tiU P * rtM i reading C6SC -ore; 8 and ylc? whUa cL 
n r i n ul r skLlls on such cescs may well represent good specula- 

seSuences Q wh}S rnerS I**™ Co . read > che neeS is for knowledge of 
sequences which come from empirical observation. 

j - ma( .. As w f observe how children learn to read or learn to 
do mathematical operations, it is clear that there are great 
f«l 1 C a ieS 1 0f sequences among successful learners. One if "truck 
for example, by the qualitative discinccion between Piaeetiln 
descnpcions of sequences in' learning or learning options and 

ized S Ss's C or f " e ^§ esCed by 8 che components' standard- 
ized tests of reading. If PiageCian observation Cechniaues were 

Xrv 1 5?ff^nf' e 2 SiVely C ?.5 ead ing acquisicion, I^e^chatT 

sueeestfd SIS tut^l* ^ d C ° nCent: for readin § treses would be 
suggested than that which we now have* 

havior ~c,Jt! e of am ? may be said about: Presumed componencs of be- 
havior, such as Bloom's Caxonomy and iCs cognitive, affective 
and psychomotor components. This popular taxonomy 'does not 
derive from observations of behavior so much as from logical 
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extrapolations from assumptions about behavior. Again, when one 
uses Piaget's designations, there are actual behavioral 
protocols, movies, videotapes, which will illustrate thd opera- 
tion which Piaget has described. I am, aware of no such lobserva- 
tional backup for Bloom and [Kraftwall] schemes. 

... T ^ us che cognitive area, in particular, looks more 

like an outline for a course in logic than a description of 
thinking. The elements in the outline may be goals or obiec- 
tives as they were originally intended for teaching, as the 
title indicates, but they are not components of thought or 
sequences of development or even necessarily hierarchical in the 
sense, that some professionals have come to think of them. 

The heart of the matter is this. We need valid 
measures of achievement if evaluation is co be vaild. If evalua 
tion also is to lead to diagnosis, we need valid measures of 
thinking which would allow impediments to thought to be identif- 
ied. However, evaluators should not allow these two things to 
be confounded. 6 

Psychology-bound evaluation. By accident of history 
and by habit, psychology has come- to dominate the field of evalu 
ation m urban education. Psychologists v^e the first behavior 
al scientists to be asked to develop an applied approach to the 
solution of educational problems. Psychologists have claimed 
the domain of mental testing as their turf. Psychologists have 
extended and refined statistical methodologies and paper and 
pencil test construction. 



However, a complete understanding of school and school 
intervention processes cannot be gained by reliances upon the 
tools of the psychologist alone. Over-reliance on limited 
psychological approaches to urban educational evalution is re- 
sponsible for the poor understanding that we have of schools and 
how they work. Psychological tests of achievement may help c P 
tell us what students know; they do not tell us whv. In fact 
they do not even tell us all the what. 

For example, we will learn virtually nothing about 
socialization of communication patterns in schools from psycholo 
gists. Anthropological methodology, such as 'ethnography and 
participant observation are but two examples of powerful data 
gathering approaches which have illuminated school practices. 

If urban education in particular is to be understood 
both the what and the why, the psychologi 1 monopoly '.on school 
evaluation must be dismantled. We must have a multidiscipllnary 
approach to the urban educational assessment. Anthropologists 
sociologists, cultural linguists and others must join psycholo- 
gists for the collaborative design of educational ""evaluation in 
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urban areas. I am aware chac we are ill-prepared for such colla 
have SS aC H P i eSenC ; WS d ° n ° C have Che disposition; „e do not 

5 ? nd ' above all > we do noC ^ve Che political 
bora?ion 8 P esSLonals • Vet: we desperately need such colla 

associatPd L wyrh X E eCt: K t:i0 K S u for children in urban schools are 

nu r ea ? 6r behavior which can be observed. The dif- 
5?f?!£"i 2 U3 HH y °l academic preparation of teachers and 
different quality of professional service for children is deoen- 

e^p ric P :Uv Ult: Th r e al ffT P ' That can be demonstrated 

empirically. The failure to handle such matters as these as 
psychological Evaluation has not, in a sophisticated wayt'in 
Ztu f L ° n ^ udLes > has ^d almost exclusively to a ghoulish 
psychological dissection of learners to locate the sou?ce of 

C?eI C £eSc°?!;;? Ce A-iJ he CreaCmenC .» differential quality in tne 
^n 3 ^ ,, I children receive, in general has escaped our at- 
5ustice.' 6 C6Chnical "Parity to remedy chis in- 

Pv fl ln«f-n^c Fin M Uy5 I WiU diS£uss cultural retardation among 

2?, f !' Many P henome na in urban schools have been invisible 
rLt T t0r \ be '-t USe ° f Che 8 eneral cultural retardation among 
them. As my brother, Thomas Hilliard, has shown, professional 
programs of preparation seldom, if ever, include the development 

on rK erC i Se 10 CUlCUr e L skiUs - Therefore, evalSSLrs ar?!^ 
on the urban scene with its rich cultural mix and actually oper- 
ate as if the cultures which they see there did not exist Thev 

thf 8 nh^ f ° rCe USe , of impropriate tools an3 concepts on " 
the phenomena and overlook the devastation that their own 
actions have created. 

d*ra on ki fiS e V here ' 1 haVe cr ied to illustrate this matter with 
data on black language, one example of which is this. The 

if? 0 ^ 0 ! of . African language antecedents to the language that 
t^i Can "^ rLCan P°Pulations speak will render certain Itandard- 
ifj?- CesCdaCa unintelligible or uninterpretable . Phonetic re- 
tention, for example, from Africa in the speech of African- 
American children is not treated on tests as retentions, but as 
pathological deviations fro^i a EuropeanAmerican norm. Quite 

6 o a Dr r y ' to a culturally skilled observer, such a norm is inap- 
propriate. Yet, no normally trained educational evaluator would 
be aware of this since nothing in his or her training is likely 
Co have provided the background for an awareness. yIc no amount 

, f sCLCa i evaluation design sophistication can correct for 
cultural retardation among evaluators. 

nrnpo , . T he evaluation of urban programs in education must not 
proceed in the absence of evaluators who are trained not onlv in 
the normal methodology,/ but in the cultural background cf those 
populations with whom chey intend to work. Such cultural train- 
ing must include history, language, and, in general, cultural 
information as well. 
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There is one Last: thing I would like Co say. It con- 
cerns evaluating what. We have spent many years looking at and 
dissecting educational failure- In that same period of time, we 
have had outstanding educational successes, many of which are 
not funded programs. Where were we as educational evaluators 
when September Clark reduced the illiteracy rate in eleven south 
em states from 12 million people to 12,000, teaching* reading? 
We were busy looking at IQ scores rather than at successful educ 
at ional practices • 

In conclusion, the specific recommendations that I 
have made are -as follows: cultural preparation, specifically 
for evaluation professionals; better descriptions of the actual 
treatment that children receive in empirical terms; empirical 
definitions of content validity for criterion measures; empiri- 
cal support for construct validation for criterion measures; 
more a[ propriate aggregation, particularly aggregation at least 
at the treatment level, for evaluation data; mult idiscipl inary 
methodologies rather than the exclusively psychological; the 
elimination of street language in professional terminology; and, 
finally, a focus^ in our evaluation on the successes in education 



PAT ALLEN, Teacher Corps: Dr. Hilliard, I am wonder- 
ing if you would expand a bit on your comment about Piagetian 
observation related to empirical data. I would like to make a 
quantum leap then to Dr. Farrar's statement about process evalua 
tion, whether that is not truly mainly a process observation, 
and mayba there is another step there. I don't know if there is 
some way for you to tie those in together or bring that out a 
little bit. ^ 

DR, HILLIARD: I guess I was appealing for empiricism 
for across the board, and empiricism and observation, not in the 
narrow sense that some people talk about, quantitative versus 
qualitative empiricism. You can have empiricism in both. The 
process observation could have and should have empirical backup. 

I was simply trying to cite, by using Piaget, in contrast to 
some of the other constructs that we work with, that his con- 
structs derive from the observations with children, and most of 
these other constructs do not derive from that. 

The same principle would be extended to a look at pro- 
cess. In other words, as I tried to say, I strongly support 
both the process and the outcomes, and they ought^ to be dene 
simultaneously or else you cannot explain the outcome. But 
where we have been short — I think that is why we' invented the 
notion of formative evaluation and summative evalaution, to have 
a legitimate way of spending time only in one camp or the other, 
where we need to be in both simultaneously. 
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So I would go for the observation of the process. 
That is why I was talking about other disciplines necessary 
to do that. Basically, the kind of data that we normally 
collect from psychological disciplines are not process-focused 
at all. That is why you need participant observation, 
ethnomethodclogies, and other kinds of things, to illuminate 
those processes. I really agree that is important. 

GEORGE LOWE, Office of School Improvement : Dr . 
Allison in Chicago was discussing the cultural bias of tests 
30 years ago. I was wondering what you think about why 
we have delayed so long when this thing was so obvious to 
us 30 years ago. Does it relate to the nature of class 
in society? What got in the way of it? 

DR. MILLIARD: Both, the nature of class and society. 

I was on a panel two nights ago at Georgia State 
and a local ETS person was there who just endorsed Jensen's 
book as the tombstone for discussions on test biases. He 
said Jensen's new book on test biases just answered all 
those questions. Jensen asserts that there is no such thing 
basically as test bias for most populations, that we all 
basically have the same environment, so there is no need 
to look at those kinds of things. m 

But, again, the problem hinges in part on a limited 
definition of test bias and a lack of empirical evidence 
on the part that they* leave out. In other words, the limited 
definition is that they restrict discussions cf test bias 
to information about item analyses, and if the item pattern 
response is the same for two different cultural groups, 
and one group is only lower than :he other but the patterns 
are the- same, the assumption is that there is no bias because 
of the statistical data. 



But, for example, there are no appeals in those 
kinds of discussions to the linguistic data. So, for example, 
if you were asking a question what is vocabulary in the 
first place, even before you do your counting and statistical 
analysis, from a linguistic perspective it would be absurd 
what we have done for the last seventy years — measuring voca- 
bulary. I think- there are about 26 words on the WAIS that 
are vocabulary words. You are going to sample vocabulary 
wiuh 26 words, and that does not even make sense in terms 
of what vocabulary is empirically form a linguist's perspec- 
tive. Those are the kinds of things that never enter the 
discussion. So we still are str k with it because of ignor- 
ance on the one hand and because, secondly, ^ome people 
want i t that way soc ial ly . 

MARIE BARPY,- Teacher Corps: Would you agree that . 
in the field of psychology we are beginning to distinguish 
between the psychometric ian and what I call* the gestalt 
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psychologist: which, in my terms, again, refer basically 
to •rhe kinds cf things that Piaget has made clear to us, 
and cnce you bridge the gap between the psychome t rician 
and the gestalt psychologist, you do get into process and 
you can begin to build criteria- such as you suggested in 
the linguistics and the logical fields that will give us 
another base for research and evaluation, including arithmetic 
once in a while, 

DR, HILLIARD: I'm glad you brought that up, 
I agree fully. The only reason that I came down so hard 
on the psychological part is because those psychologists 
who are most closely associated with program^ evaluation are 
not the gestalf psychologists • They are from another crowd. 
Again, I am not even putting down that part of psychology 
because I think, quite clearly, that each part has some 
part to play. But the domination and exclusive participation 
and control by that crowd over evaluation I think is killing 
us , 

MS, McKENZIE: Dr, Gold is Senior Research Associate 
at NIE and has conducted a number of evaluations of urban 
intervention programs, 

EVALUATION METHODS APPROPRIATE FOR INTERVENTION PROGRAMS 

BY NORMAN GOLD, 
SENIOR RESEARCH ASSOCIATE 
TEACHING AND LEARNING, NATIONAL INSTITUTE OF EDUCATION, 

I want to spend just a moment changing the title 
of my presentation to "Evaluation Methods Appropriate for 
Intervention Programs," rather than "effective methods," 
The reason I want to do this is co make sure that there 
were not a set of expectations for what I have to say that 
I cannot meet, ■ 

I . ■ 

By way of introduction and orientation, I was 
looking at a picture in "The Washington Post" a few years 
ago that really struck me and stuck in my mind for a long 
period of time, until this moment. It was a picture of 
a -an on a sailboat, heading south, down the Potomac, east, 
• then south, to the Caribbean, He had his family, dog, just 
about everybody on the boat with him and he was leaving, 
A reporter ^asked him, "Where are you going, and why?" He 
said, "I'm going to the Caribbean ana am taking my family 
with me. The reason is because I either had to accept myself 
as a failure in terms of my set' of experiences, or I had 
to redefine success." ^ ^ 

^ \ It seems to me that with the literally thousands 

of urban intervention programs on the landscape, all of 
which have been determined as failures, there are possibly*, 
alternative explanations. either have to accept the 

fact that all of the effort and all of the energy over the 
last 15 years is. in fact, a failure, or we have' to redefine 
the measures, that we used in terms of what we mean by failure 

er|c * 26 
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or success. It occurs Co me that a possible explanation 
tor this dismal record of intervention programs could be 
attributed to unrealistic expections of what can be achieved 
over a given period of time. - 1 

Each program that is formulated and supported 
for development promises a direct and impressive impact 
upon the set of problems for which the intervention strategy 
was defined. Students will perform better, will feel better 
and will act better as a direct result of the support exper- 
ienced as participants in the program. These effects also 
will be sustained over time. Teachers and administrators, 
community participants will embrace the effort and continue 
implementing the program on their own. This effect will 
occur m some impressive degree within a specified period 
or time, and the program will be implemented, students will 
be processed through the program and will' come- out on the 
oflier side better. 

Described in this somewhat irreverent way it 
also occurs to me that one of the characteristics of these 
expectations for intervention programs is that they are 
somewhat similar to wishes. The problem as with most wishes, 
is that they often have limited contact with reality. This 
gets rather serious when we incorporate these wishes into 
a tormal set of expectations, for example, in the form of 
government contracts and grants. If „ e have learned anything 

..MV; 15 yearS ' iC is that the wishes and promises 
that initially, gave us hope can come back to haunt us. 

We raise expectations of a wide range of people, including 
/those we serve. We place excessive pressures on service 
providers -nd recipients. Probably one of .the more tragic 
resuUs is that we really have little opportunity for putting 
together all of the ingredients to mount significant locally- 
based intervention programs. ' 

As far as I can see now, the opportunities will 
decrease for these kinds of programs over time. When expecta- 
tions are not met, the individuals associated with the efforts 
iose credibility. The amount of investment individuals 
are then willing to make in the future diminish. It is 
my opinion that the disillusion and skepticism regarding 
social action intervention can be attributed partially to 
unmet and often unrealistic expectations. It has been my 
experience as well that the multitude of negative evaluations 
ot urban intervention programs often reflect this discre- 
pancy, between the wish for program effects and the reality 
ot the complexity of the problems addressed. 

I think most people would agree with the dismal 
record that I have mentioned. We either have to accept 
the fact that we have failed and these failures are associated 
with a continued need to create realistic expectations for 
intervention programs, or we must redefine success by accepting 
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more modest gains achieved over Longer periods of investment. 
At this point in the development of evaluation, I think perhaps 
the most appropriate role evaluation can play in social 
intervention programs is to contribute its expertise in 
efforts to forumulate more realistic program expectations 
and implementation strategies. About five years ago, I Started 
to take this issue seriously at the outset of evaluations, 
^rather than as customarily done when the evaluation is com- 
pleted. To assert that expectations for a program were 
unrealistic when it was over was a benediction that most 
of us were not very comfortable with. 

T will now describe a set of operations and char- 
acteristics of evaluation to support the program development 
process. These are: modifying stakeholder expectations; 
developing inductive and dynamic evaluation strategies that 
are compatible with the development process; moderating expecta- 
tions for evaluation; and measurement strategies for 
developmental programs. 

_ Given the need of program developers and program 

runders to over-promise, it is likely that initial expecta- 
tions will remain considerably optimistic. It is also true 
that our ability to predict at the outset either the course 
of program implementation or effect is still severely limited. 
Appropriate evaluation should assume a large share of the 
responsibility of providing information useful in grounding 
expectations. As with our friend the sailor, the adaptation 
process is to either change what you were doing to meet 
a set of expectations or to change your expectations. This 
process is necessitated by what is often interpreted as 
failure; but, in fact, it is gaining insight and experience. 
The evaluation process employed to aid in developing more 
realistic expectations is through successively approximating 
what is possible, given a set of resources, a period of 
time, and an increasingly elaborated conceptualization of 
the problerr being faced. 

A first task of evaluation is to negotiate an 
initial set of expectations that are fairly well rounded. 
This is done by piecing together representative expectancies 
from a variety of sources. Utilizing a variety of sources 
is important to avoid representing a single set of interests. 
The stakeholder group is designed to obtain expectations 
from all parties having a direct interest in .the outcome 
of the evaluation. These groups represent federal, local, 
programmatic and participant interests. They are asked 
what they expect the program to produce and in what time frame. 
These sets of expectations are characterized by tvpe and 
interest group. 

The evaluation process then matches these expecta- 
tions with both the proposed program strategy, the Drogram 
development history, if any, and the history of efforts 
ot this type. This provides some check on the reality of 
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expectations. The next is to determine what realistic form 
of evaluation perspective the existing programs can sustain: 
the capacity of the program, the time for the evaluation, 
the sources for evaluation, and the caoacity of evaluation 
technology itself. 

The initial evaluation approach emerging from 
this process as recommended to the stakeholders by the evalua- 
tion then consists of a set of expectations attached to 
a. proposed- or actual set of strategies that can be systema- 
tically observed during the life of the study. This proposal 
then is negotiated with all stakeholders and the evaluation 
proceeds only upon significant agreement on this or some 
acceptable alternative approach. This process can take 
rrom six to ten months to accomplish. 

What is achieved is an evaluation based on a set 
ot initial expectations that are somewhat rounded. More 
importantly, -these expectations are shared in common over 
a range of vested interests and are viewed as preliminary 
and developmental. J 

..... '. There are many factors that will determine whether 
this initial expectation requires modification. Among the 
most important is the rate and extent of program implementa- 
tion. Ine initial preoccupation of appropriate intervention 
program evaluation should be focused upon program implementa- 
tion. Though some effort at obtaining base-line data for 
later impact assessment may be started, the major focus 
upon which all else will depend is the program implementation. 
ic is otten tne case that program implementation will be 
more complex and take considerably longer than -expected . 
It is also the case that to promote local involvement, consider- 
able variation m program operation will occur. Finallv, 
as knowledge of the complexity of the problem increases', 
it may be determined that expectations for behavioral change 
were too ambitious and more modest changes that proceed 
or function as proxies for more elaborate outcome measures 
may be more appropriate. 

This type of evaluation ought to be a part of 
an annual process of reappraisal to determine if the rate 
and direction of the program being developed is as expected. 
It significant discrepancies exist between expectations and 
what is observed, then modifications in program strategies 
or goals and objectives ought to take place. This process 
of program modification should continue until the program 
stabilizes to a point where those concerned feel comfortable 
contortaole with the way the program is operating and with ' 
what is expected by the vested interests. 

To enable this to occur requires explicit acknow- 
ledgement and support of an iterative process' of program 
development from all parties directly involved. Though 
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this process in fact: has operated in some programs, for 
example, in Headstart , it: is enabled by political pressure 
and manipulation by program advocates. If more appropriate 
program development is to take place, serious rethinking 
of program development process needs to occur. 

If we accept as at least a partial explanation 
of the observed problems in intervention program development 
the notion of reality testing and iteration, then our notion 
of the evaluative process must change dramatically when 
applied to this context. It is my view that the evaluative 
process in this kind of setting needs to be inductive and 
dynamic . 

It is inductive in the sense that it gains knowledge 
through observing actual events. The event is the teacher. 
You learn what it is over time rather than matching the 
set of events against some predetermined set. The goal 
is to know the program as it actually occurs and to under- 
stand and describe as best you can why it has evolved in 
the way that it has . 

It is the actual comparison between this process 
with expectations that are held by the population of stake- 
holders that allows the discrepancy notion to exist. It 
is not discrepancy as normally applied in evaluation. It 
is not discrepancy in terms of a pre-set notion of the program. 
It is discrepancy between expectations and what the program 
actually is, what has been observed. 

The evaluation under these kinds of conditions 
needs, as well, to be dynamic, because you have to allow 
for the fact that programs can change and adapt rapidly 
and radically. If you set up a whole set of data collection 
structure, either it has to be so wide and expansive that 
it is almost too weighty to continue, or you have to be 
so specific that you miss a great deal of what the program 
actually is doing. Descriptive techniques, rather than 
large precept data bases, are much more powerful in this 
kind of environment. 

The goals for evaluat ion need to be moderated . 
Unreal i s t ic goals and expect at ions are not jus t the domain 
of program developers and program supporcers. We expect 
and have been told that evaluation will provide the definitive 
answers as to whether programs have been effective or not. 
Those answers also will be delivered within the time of the 
evaluation contract. The complaints about evaluations are 
that they are often equivocal, the findings are obscure, 
they are inappropriate for decision-making, they are poorly 
timed and they are not readily obtainable. This is the 
data which sort of gives us the feeling that there is a 
gap be tween expectations and reality, as far as eva luat ion 
is concerned. 
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~e S °' whaC is reas °nable Co expect: given the state 

of Che arc aC present? WhaC we can gain from evaluaCion 
iL understanding of Che magnicude and nacure of 

Che problem I chink chac has been very definicely and 

ItlLt S K 1 K ed . by ^ Sa Hilliard - By understanding che problem 
better, by having beccer explanacions of che problem this 

' important information Co program developers and inno- 
whv rhiv 3U K e ChSy Can begin C0 understand and help explain 
«hL ing P roblems in certain kinds of areas, 

what they are missing, what they are not seeing, what kinds 
of adaptations or alternatives they might try. 

mant . We can use evaluation to aid in the program imple- 

mencacion process. Much of che Cime and effort of che 
evaluacion people who work on projecCs chac I fund are spenc 
in understanding the implementation proces and providing 
information that is very useful in the iterative development 
of programmatic sCraCegies. We can conduce analysis ChaC 
DarcIciL e nr^ de H whaC u c °ndiCions che program facilicaTres 

EiS 2£? £ a C d ,, in w ^ ac way - This is ver y much ^ "n« 

wich whaC Dr. Hilliard said in cerms of sub-group analysis. 

Wh f^ we . are noc parcicularly interested" in is 
great generalizations. All of our strategies have been 
designed in the past so that we can generalize what we are 
finding out. Then we find out we don't know anything aSd 
so we have nothing to generalize. The statistical techniques 
Chac we used, che parcicular designs are orienCed toward * 
™k Ki zaCl0n ' In che Program developmenc conCexc, chis 
probably is noc very useful. WhaC we can do is Co look and 
discover for Che parcicular program aC which we are looking 
where is ic facilicaCing behavior and under whaC condicions. 

Anocher problem is chac we have Co finally rid 
rilh'lZV'n^tl" ! S develo P m ent a l programs are concerned, 
wich the cost/benefit mentality. It is the cost/ben-fit 
mentality that keeps us having to find major effects. 

—i * - u If we J can acce pt partial answers from several 
M«n & !£ ^"dependent, efforts, we can provide the founda- 
tion for mafe effective efforts in the future. Beyond the 
best case analysis procedures to which I have alluded are 
two ways cross study learning can be facilitated. The first 
is Co break down parochial program inCeresCs, che nocion 
Chac each program has Co solve che major sec of problems 
Co gee supporced. A program should noc have Co promise 
cocal answers Co complex problems, buC is valuable if ic 
can make a conCribuCion Co Che knowledge base chac we already 
have. Secondly, we have Co concinue Co encourage cross 
program synthesis of programs that deal with the same popu- 
lation working with the same kind of problem. 

ant . c . L ? C me address some comments on appropriate measure- 
ITIJ * intervention programs. First of all, it seems to 
me that we just ought to get rid of the set of things that 
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drives us in terms of generalizations. What we don't want 
to do is necessarily generalize, but we want to discover 
jhat works under what conditions. Programs have to be willing 
to be held responsible or accountable, that is true, but only 
for what they have some ability to manipulate directly. 
It seems to me thac mostly they can be held accountable, 
for government purposes, to improve the provision of some 
service that they are providing. The measurement of " this 
service is that it is useful to service providers and faci- 
litators and it is attractive and useful to clients. 

As far as finding out what happens to people as 
a result of the program, I think what we need to do is con- 
struct broad categories of behavioral changes that might 
be observed. Those broad categories ought to have in them 
a wide variety of possible behavioral indicators. I will 
give you an example. One set is a notion of investment' 
behavior. Investment behavior, is that set of behaviors 
which a program may affect which causes a student to be 
able to invest more in the educational experience. The 
sorts of programs with which I deal are very much oriented 
toward increasing this investment. 

There are a whole set of indicators to show that, 
in fact, this investment is occurring. Attendance is invest- 
ment. Doing work" is an investment. Getting along in the 
particular social setting is an investment. Being less 
isolated is an investment. The point is that we don't know 
nor can we be responsible for the specific behaviors that 
people will adopt or modify, 
i 

I think one mistake that we have made is that 
we have had to promise that if we do this, these are the 
behaviors that will take place. I am concerned about that 
as a general issue. I am concerned that what we do is to 
place a conceptual and measurement straitjacket on service 
providers and recipients. If you are operating appropriately, 
then you will operate by doing these things, and if you 
are reacting appropriately, you will start doing these things. 
That's why I think what we really need to* do is to have 
much broader categories with a range of possible kinds of 
effects that could occur and to take this measurement strait- 
jacket off. 

Wherever possible, use observation of behaviors 
rather than standardized tests. It seems to me that there 
are tremendous amounts of evidence that are bui lding up 
all over the place and that have been building up for long 
periods of time showing that our penchant for scale measures 
just will not teach us very much. Given the kind of. vari- 
ability that I have described, observation seems to be much 
more powerful . 
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The type of evaluation procedures described in 
this presentation places evaluation, squarely in the program 
development process. The notion of formative versus "summative 
evaluation is., obscured . Evaluation contributes in ways 
it knows how by testing discrepancies between expectations 
and observations. It also documents the history of the 
_„J.d±Y^^ Both of these qualities are 

extremely useful for learning and contributing to the adjustment 
process. The strategy is to reduce the adversarial notion of 
both program intervention and its counterpart, program evalua- 
tion. The intervention is designed to facilitate local efforts 
at improving practice, not to supplant those efforts. 

The goal of evaluation is to help in the program 
development process rather .than judge. Judgement is held 
to a minimum. This is appropriate since evaluation, as 
we know it, at least at this point, never does very well 
at this job anyway. Final judgments usually are made on 
a whole array of factors in any event. Evaluation thus 
is cast more in a positivistic framework in an effort to 
contribute vigorously to the opportunities for more success- 
ful program development in the future. 

SHIRLEY JACKSON, Director, Basic Skills Program: 
My question is for Asa Hilliard. It has to do with items 
discrimination as a vehicle for test bias. I want you to 
comment on 'this in the development of national standardized 
. tests, the potential that is inherent in the whole process 

of items discrimination as a vehicle for test bias. I don't' 1 
think that has, been looked at too carefully. 

DR. HILLIARD: Very briefly, item discrimination 
as a vehicle for test bias is usually duscussed most in 
the area of IQ testing as opposed to achievement testing. 
It is most appropriate, I think, to look more closely at 
it with IQ testing because it helps to determine whether 
or not the thinkings that are supposed to be .measured, the 
behaviors that supposedly are being tapped, are really being 
tapped. ^ But to limit the focus in that instance of the 
discussion on item discrimination and eliminate the rest 
of the information that we need to me is unscientific. 
' It is unscientific because empirical data in other areas 
relevant to that question are completely left out. 

Now, people also look at the patterns of groups 
on achievement tests, and there are some instances in which . 
there are cultural variations. That is what I was trying 
to illustrate in the area of reading tests, in particular. 
Here the pattern analysis would not be sufficient to determine 
whether an item was bias'ed. But what we finally determine 
we are going to do about it depends on the use of the ach i eve — 
ment test. If the use of the achievement test is to measure 
whether people have come up to a norm upon which we have 
agreed, then I am less concerned about the issue of bias 



33 



Page 32 



in the achievement: test. But if the achievement test is 
used in a diagnostic fashion, then the whole issue of bias 
becomes much more prevalent in my mind. I could say a lot 
about it. As a matter of fact, I have a whole chapter on 
it coming up in about two week. 



GUY SHEFFLER, Teacher Corps: Dr. Hilliard, in 
your speech you made one remark which I would like you to 
cl arify. You said that there were some teachers who got 
along with the troublemakers. My question is how do you 
cl arify troublemakers in that fashion? Do you clarify it 
as being learning disabled, or how? 

DR. HILLIARD: Well, I don ! t think I used the word 
"troublemaker What I was referring to is that there -are 
some children who are thought of as hard to teach. Usually 
we are talking about poor brown kids, poor white kids, and 
people who go into those communities frequently believe, and 
I think we see abundant evidence, that nothing much can be 
done for these children. v 

One of the reasons we believe that is because we 
do not look at what is being done under certain circumstances. 
In other words, I was saying that not all teachers fail with 
the children that we have identified as unteachable. When 
I say "not all," I mean that there are thousands, literally 
thousands of teachGrs whose performance is obscured by the 
way we collect information. Thi-s leaves us with the feeling 
that no one is doing anything because, on the average, not 
much seems to be done. 

One of the ways I have tried to deal with this is 
to collect both anecdotal and program information on successful 
educational innovation. It is similar to some of the things 
that Ron Edmonds^ftas done. I don't see how anyone can look 
at Edmonds 1 work or the work of some other people, such as 
James Comer who have seen things turned around with the unteacli 
able, and can continue to have low expectations for the child- 
ren rather than to focus their attention on the intervention 
that they are not receiving. 

I don't even call the children troublemakers. I 
think troublemaking is a sympton of the fact that they are 
not getting attention. Those same children have long attention 
spans and are completely involved and need no external discip- 
line when they watch TV. But there is a good show there. 
They don f t have good shows where they are. 

MS. McKENZIE: Thank you very much. You both gave 
us a lot of insight into appropriate methodologies for evalua- 
ting urban intervention programs- My point of view is that 
it is surprising that Norman and Asa were not not so very 
far apart in how they approached this topic. 
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DR. HILLIARD: I have Co make one observacion. 
If.JL_understand Che basic pare of what Norman was saying, if ic 
was focused on unrealiscic expeccacions for some programs, in 
other words, if chere are programs chaC really were sec up Co 
do limited things and we evaluate those programs as if they 
we^suppjDsed to be tak-.ig care of ocher Chings, Chen I don't 
have much problem wich chaC. BuC if che unrealiscic exoecta- 
Cions are ChaC programs should noc be expecced Co change Che 
outcomes for children, Chen I would have a lot of problems with 
that because if anything, I want evaluation to raise the expect- 
ation. This is why I gave the example I did in the begin- 
ning. ' * 

I don't see how we can sit here in America and look 
at the progress in the Soviet Union, look at the progress 
in Germany, look at the progress in Japan, look at what they 
do with urban educaton, and not realize questions about why 
we are so far behind. I think it is time now to jump anything 
that does not tell us how we can change that gap. I am never 
going Co be- sacisfied wich any explanaCion chaC suggescs ChaC 
ic is in che genes of che kids. I have noc been able Co figure 
out how chey could deCermine che genecic difference beCween 
Russians and Americans. 

MARSHALL SCHMITT, NaCional Diffusion Network: I also 
think we have to be careful about taking statistics ^rom other 
countries and making a comparison with kids in this country in 
terms of proposed numbers of kids and che cypes of kids in che 
Soviet Union and elsewhere, comparing chem Co Che general kids 
ChaC we have . 



DR. HILLIARD: Well, I have Co say someChing on 
ChaC. I know that chis is someChing people someCime say. 
The way we oughc to determine whether or not we can say that 
is by empirical observation. We' don't have empirical observa- 
tion of what happens when adequate treatment is provided to 
children. That's why .1 say we never have described adequate 
treatment . ^ 



We do have empirical information. I did not take 
the time to list all of these, but if I had to, I could. 
For instance, there is the Oakland Community School in 6aKland, 
California; there is the Marcus Garvey School in Los Angeles; 
there is Marva Collins' school in Chicago. There are just 
any number of times that people have gotten fed up and have 
decided to go and do someChing different with these kids. 
In other words, there is nothing wrong with these kids. When 
we say that we s ispect that there is, we are not dealing 
with empirical data. We are dealing with what happens to 
these kids when they don't get a good program. We are looking 
at a program which has been terrible for them. 

MR. GOLD: Let me address this for a moment. I 
said nothing about expectations for individuals in my remarks 
and they should not be interpreted in that way. I really 
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said nothing about, expectations for programs. What I said 
was that expectations had to be somehow related to what you 
ate doing, the actual events, the actual set of operations 
that yoru undertake to make those ' changes . What my experience 
has been is that there is tremendous discrepancy between rhe- 
toric and reality. My feeling is that it is not a very con- 
structive ro?e for evaluation to simplv document that discre- 
pancy. My studies are continually interpreted by the press 
as showing that things do not work, and tney keep reflecting 
this discrepancy. 

What you then have the option of doing and what 
I think is an important contribution that evaluation can make 
to program development is you have the option either of chang- 
ing what people expect or changing what you are doing so that 
you have a chance of meeting the expectations. 

Is that clear? It might not have been clear a moment 

ago. 

MS. McKENZIE: I think we will have to ponder it 
for a while. Let me make one footnote with respect to the 
comparison of our youngsters with the Soviets, West Germans, 
and Japanese in science and mathematics. There will be a 
report, I think issued from the White House, very soon on 
a study of that situation. I think we ard going to have to 
lock at it very carefully because basically we do have low 
expectations for our youngsters in those two areas of study. 
It will serve to our detriment and to their detriment because 
we are not keeping up with technology in our way of education 
and providing young people with a notion that learners should 
be dependent on educators rather than moving to push learners 
to be independent , using all the strategies available to them. 
So, with respect to poor youngsters and minority youngsters, 
and the youngsters that d© not get adequate support frcm fami- 
lies or communities, we do need to push and to examine what 
we are doing with them, part icularl ly in the areas of science 
and mathematics. 

PANEL DISCUSSION: 
ESSENTIAL ELEMENTS FOR URBAN EDUCATION INTERVENTION PROGRAMS 

INTRODUCTORY STATEMENT BY KATHLYN MOSES, 
DIRECTOR', URBAN INITIATIVES, 
OFFICE OF SCHOOL IMPROVEMENT 

Each of the panelists is himself or herself a 
principal speaker, a keynote speaker. So to ask them to 
serve on a panel took a lot of n hutzpah. ,f The next point; 
is that you may look at this panel and think that you have 
here a Mulligan stew. You are absolutely correct. We have 
the press represented; we have research represented; we 
have a large public school system represented; we have a 
suburban school system represented; we have a classroom 
teacher represented; then, to really put the icing on the 
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! 6 ; h W have a recipient of all of our responsioilities , 
?n a c student, who will evaluate the program, not 

,,?h*n T ClflC . Way buC from Che P° inc of vieS of what an 
urban intervention program has done for her. It ray be 

'his fe irthe fr bottom y ISe earCh y ° U WlU ™ d ^ ~> ^ 

JAMES P. COMER 
MAURICE FALK PROFESSOR OF CHILD PSYCHIATRY 
VALE CHILD STUDY CENTER, 
AND ASSOCIATE DEAN , YALE^ UNIVERSITY MEDICAL SCHOOL 

t h ' Ms r M <;Kenzie had trouble defining what I do. 
I have trouble defining what I do as well. I do not consider 
myself a researcher, unless it is an intervention researcher 
or an evaluator, and I kind of cringe when I am c^lleTs^ch!' 

of m „ ™ 1 Cl ? ink 1 ? oC carted in this business because 
bftvLS a ? x P erience in looking at the discrepancy 

between the performance of my low income classmate on the 
playground, in church, and elsewhere, seeing that thev were 

HI w ng ^ Tn\ P h t r o°ng Tun' ' Cl ^\^hin g 

f-hic * ee rL aT u g run ' Led to my concern about 

b Llf S h' becoming a clinician eventually and with 
as P an l-r^tl ^background it seemed reasonable to be involved 
as an active participant-observer in the school program,' 
because it appeared to me that there was an oppo?tSni?y 

rn XF K C t? P Finci P L " V £ Che social and. behavioral sciences 
to the problems in school. ei • 

We did' exactly what Dr. Farrar said in our school 
program in New Haven, where there was a dramatic improvement 
in achievement and behavior among students in one of the 

around 51 filJ n f pe " f ° Fmance - u We hepped in and messed V 
coSid nor L ^ C u m the ™ v y beginning that the problem 
could not be with che students, because, in my heart, I 
knew from my own experience that they had the ability. 

hoi™ n - CL ^ arL ^' my concern about formative evaluations 
being imposed and rigorous research projects baing imposed 
on systems shows that there is so much that must be done 
before you can do any kind of evaluation in systems. We 
IS LS?^ 1 ' 33 researche ", generally. The intervenor 

' abhs P rusc amon § the people involved in the system, 
and that can take anywhere from two to five years, depending 

they'd y °V re ' righC ° ff Che baC ' You ha ^ to'deal wl h § 
has of ^'"h 3 ; y J C ^ A SySCem ChaC is down over time 
it it Z ?u ad W ^ od ' and - vou know ic - To pretend that 
it is not there is dishonest. You have to go in a'nd you 
have to deal with that dead wood and at least get it out 
° r h ^, wa y or § ec ^ to a point where it cannot interfere 
with the program. Your have to deal with opportunistic 
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people and manipulators. You have to deal with the press, 
which wants to destroy you. You have to deal with all of 
the doubters, all of the people who are" there after you. 

Now we know this. We cannot go into systems and 
pretend that we do not have to do all of those things, 
Ytfu have to do all of those things before you can permit 
a staff to systematically address the educational issues 
the treatment programs that you are going to provide for 
children in a program. That is part of the process of inter- 
vening in schools. 

Also, the implementation of the program and looking 
at the implementation is important. It seems to me that 
in order to intervene in a school system and implement a 
program, there are several things you must do. You have 
to have reasonable intelligent intervenors in the first 
place. They don't have to be brilliant, just reasonably 
intelligent. You have to have people who have reasonably 
good interpersonal skills and relationship skills, I see 
people trying to intervene in schools who cannot get to 
first base with the people with whom they are dealing because 
they themselves do not have good interpersonal and good 
relationship skills. You have to have people who have some 
knowledge of human behavior, of system behavior, and of 
what Dr. Hilliard talked about, the cultural style and ways' 
of the community, 

'But you also have to understand what is a by-product 
of trying to function and operate in a system that is essen- 
tially hostile to your goal and what kind of behavior that 
promotes. That is a large part of the problem. When I 
listen to people talk about parents who do not want to parti- 
cipate, a large part of that problem is related to the inter- 
actional problem between the groups who have been denied 
the system and closed out of it and the people who are iri 
control of the system. You have to have that knowledge 
of the history -and the relationships that Dr, Hilliard talked 
about , 

Finally, you have to have people with good problem- 
solving and/or management skills, people who are capable 
of establishing goals, of establishing strategies and of 
developing mechanisms in which you can develop specific 
programs or treatment relative to the needs in that particular 
school. You need people who are able to measure the behavior 
changes and outcome changes within a school, who are able 
to identify the problems that they see as a result of the 
data they collect. 

Let me give you one specific example to try to 
concretize what I am talking about.. We have a learning 
center in our school program. The learning center people 
went in and worked with children who were supposed to be 
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retarded. They discovered afcer cheir program thTt. che 
children who were supposed Co be retarded were coming out 
scoring higher than the so-called ^normal" children in the 
tirst and second grades of the school. Well, something 
obviously was wrong wich the program being given to the 
fhlm ?f n °™ 1 " children if che others were out-performing 
Sa?Vi had had r ^ id > separated, rigorous programs 
that did not look at. the total school, we would not have 
round that outcome and we could not have pinpointed the 
problem and helped the staff to see that the problem was 
theirs and not the problem of the children. 

In order to work in that wav, it seems to me that 
evaluation has to take all of the issues that I have iust 
^"^? ned u f nC0 acc °unt — the style of the intervenor, the . 
scarr s abilit>, the knowledge of human relations, system 
behavior, cultural and interactional problems. But/in 
addition to this, we have to consider as we evaluate programs 
the stability of the staff in terms of staying power. When 
you are always turning over, always starting over, then 
there i : a very serious problem because you will never move 
forward. 

k n tP 6 raCe and direcC ion of change, as mentioned 

Dy Dr. Gold, is tremendously important to look at as we 

do evaluations. There also are some other things about 

which we never talk; and yet, as policy-makers and researchers 

we know that they are tremendously important. These form 

the power issue, the control issue that go on at every level 

within a community, within a school, among staff, between 

start and principal, between staff and support staff at 

the central office level and at the state and national level. 

All ot those issues are at play and we have to pay attention 

to them. 



I agree very much with Dr. Hilliard's comment 
that m order to pay attention to those and other kinds 
or issues that are very troublesome, we need different ways 
of doing research. The ethnographic studies, the participant- 
observer approaches, and the case study methods used by 
business I think can contribute much more to our knowledge 
ot school intervention programs than some of the kinds of 
approaches that we use right now. 

Piaget was mentioned. I want to point out that 
Piaget was a clinician. Piaget asked question as a clinician 
would ask questions. One of the problems with our studies 
and one of the problems with our use of questionnaires is that 
when we ask questions,, we don't know whether we are asking the 
right questions. We don't know whether the respondent is 
answering the question we thought he thinks we are asking. So 
it is very difficult. With the approach that Piaget uses, you 
can move from one question insight to another question. This is 
much more helpful that the approach we use right now. ' 
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I Chink that: in order to look at something as 
complex as a public school system and the social system 
within an individua . school we have to use &n ecolpgical 
approach or an ecological perspective* We have to examine 
each element within the qystem — parents, teachers, admini- . 
strators, the young people, their training, their development, 
their capacity to do the job they say they are going to 
do or want to do, the kind of support they are receiving 
to do that particular job. Each element of the system must 
be looked at and not just the job. 

I think that a second element of an ecological 
perspective would be that we have to look at the process 
ahd content of the input or, as Dr. Hilliard said, the treatment 
that the group uses. Is the intervention conceptualized 
correctly, based on what we know about the children, the 
school, the power issues, and so on? Have we prioritized 
what is important? 

Someone mentioned that we have to look at the 
services. I .think in a system that is chaotic, ycu can 
do almost anything and have some kind of "good outcome. 
But is that good outcome what you really want? We have 
to say what is it that we want and is the treatment designed* 
to bring about what we wart; and, even if it brings about 
what we want, is that relevant to the major issue, which 
is improving the academic and the social performance of 
the young people. We have to ask that question. I personally 
feel that too ,many programs are funded to achieve frivolous 
goals, and those goals are achieved, but that is not essen- 
ially what we are after in trying to improve urban education. 

Finally, do the intervenors have good timing and are 
they aware of timing and are they able to respond to the timing 
issues? Are they opportunistic? Are they aware? Do they watch 
for the changes in administration downtown, the changes in 
rules, the people who move to different positions who can be 
helpful and usetui? Are they political, in other words? Now 
those may be dirty words to researchers, but those of us who are 
muddling around in a messy area are concerned about these kinds 
of things. The kinds of things we try to do in the first year 
of our program we could not do if the person in charge resented 
our being there. But when that person moved off, and sometimes 
they were helped to move off, we were* able to do some other 
'kinds of things. Those are the issues. That is the way school 
systems work, and we have to be responsive to that as we try to 
evaluate school programs. 

Finally, the interaction of the various elements 
within a school system, the people who are involved there, 
parents , teachers , administrators and children, and how they 
interact all have to be lookec at if we want to determine 
the effectiveness of a program. Interaction is tremendously 
important, and yet, we pay very little attention to it as 
we* try to evaluate programs. 
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In conclusion, I would jusc like to sav that the 

tH kLd°orki Ua ?T "r arCh 1 * ink '""^res'w cn achieving 
Che kind of knowledge that can be useful. I chink chat s 

a „^ Ve C ° C f \ an a PP r °"h Chac examines every aspect of 
wich fach ocher h if" ay Ch * ° £ a P ro S ra ™ i«eracc 

to bs done uirM n "5? r f ally to det e™ine whaC needs 
we are able Co ^""J" SCh ° o1 s y stem and " ha ^her 

C r ocl? ievel SP0 ^ a ofea C co Cnol f ^"-"l" 

RAMON C. CORTINES 
SUPERINTENDENT OF SCHOOLS, PASADENA, CALIFORNIA. 

u . 1 am here Co Cell you that we can educate HHc 

whatever their status or lack of it, whatever "heir color 

l believe this to be true. We can educate kids and we can 
do so, whatever their backgrounds H^nir* t-hUL I- 
which scaci<?f-ir a n» — ^ j * ' des P lCe Chose circumstances 
wn L cn statistically may tend to accompany poor educational 

S2°S B s:u.S d fS 1 2. m 311 co ° ea ^ and - s-r 1 

been .{yrfS^VS K^ f ^TK£i District 
schoo d?s r tr? c r a 23 '°?° SCUdenC microc °^ of a modern urban 
alteratives and ?™i f" 8 * e ?° Ugh C ° P rovide educational 
,n • 5 SmaLL enou S h Co manage them successfully 

olec/a^a^ 

r-nro= C h y a-|ec^n n f JSSfc since* if ¥ ' 

28 percent decrease. Last year, close to 40 percent of 

?s f e^^nce^nat'anv'r-f 3 ^ 1163 reC6iVi ^ AFDC "anf there 
are fearful o' LnlvL* f ^ P arCi ^ La rly Hispanic families, 
Nearlv hfif f a PP L y in g for lC , even though they qualify. 

a L n ,° U h r SCUdenC i ive wiCh only one parent or 
percen? whftf H T g ° ne fr ?7 a , district which was 66 
in 1966 ? Percent black and 6 percent Hispanic 

in 1966 to one which last year was 32 percent white 44 
percent black, and 20 percent Hispanic. ' 

Rowi M Q In v Sh °f C '^ t:he Pasadena y° u see behind the Rose 
Bowl on New Year's Day, while not a painted backdrop bears 
AWU resemblance to the community served by the schools 
h n t, " S^graphy we often are considered a suburb of' 
Los Angeles, Pasadena, by history and circumstan-es is 

an3 U K LnC ; S SS T iaUy Urban commSnJCv in its own rUht 
and beset by the problems of nearly eV erv American fi^ 
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From court: ordered desegregation Co Che budgecary prdblems 
of recenc years following Proposicion 13, and in che pace 
of inflacion we have had Co adjusc our efforts and oyfr programs 
Co accommodaCe ic, and I believe successfully so. ( 

Lec me illuscrace wich respecC Co our disCricC^s 
overall performance wich respecC Co Che performance of our 
compensacory educaCion program funded by ESEA , Tide I and 
ics California equivalenC. Since 1974, che disCricC's 
educacional performance, as measured by che comprehensive 
cesc of basic skills adminiscered Co all sCudenCs each spring 
during che week in which Che cesc was norm has increased 
dramaCically from che 45ch percencile in 1974-1975 Co Che 
53rd percencile in our spring, 1979 cesc battery. Over 
the same period, the English language test scores of our 
Hispanic students increased from the 35th percentile to 
the 45th percentile, while the scores of our black students 
increased from the 27th percentile to the 42nd percentile. ' 
All of these gains, gains in terms or national norms, occured 
at a cime when Che disCricC's populaCion was becoming in- 
creasingly dissimilar Co Che norming group — poorer, blacker, 
browner and all of che ocher excuses. Meanwhile our whice 
populaCion maincained ics performance consiscencly becween 
Che 65ch and 72nd percencile. Preliminary resulcs for che 
1980s indicaces chaC chis performance across echnic groups 
has been mainc.ined and overall che disCricC is now aC che 
54ch percencile. 

Gains in our comp ed program, a program in which 
I have been deeply involved and of which I am mosc proud, 
have been perhaps our mosc impressive. For example, in 
alongicudinal sCudy conducCed by our planning deparCmenC 
of research and evaluaCion, che Tide I program was assessed 
in Cerms of academic performance of a maCched seC.of program 
pardcipancs on Che CTBS as Chey passed Chrough our primarv 
K-3 schools becween 1975 and 1978. The resulcs .were examined 
in reading, language, and maChemadcs in cerms of median 
percencile scores Co make reladve comparisons wichin che/< 
disCricc becween schools and agalnsc che nadonal norms. 
Only chose sCudenCs who were CesCed in che disCricC on all 
tour cescs, for example, each spring from 1975 Chrough 1978, 
in ac lease one subjecC area, was included in sCudy. 

If we are Co consider che face chaC che cesc pub- 
lishers' norms are based upon che assumpCion ChaC all Chings 
remaining equal, a child's or school's percencile score" 
is expecCed Co remain Che same in successive years, Chen 
increases in percencile scores represenc greater than expected 
growCh. In che case of Che Tide I program, such growCh 
can be seen Co represenc evidence ChaC all Chings have noc 
remained equal, ChaC, in face, compensaCorv services have 
oeen provided and chaC che funds have had Che effecC rhev 
were incended Co have. 



9 

ERIC 



42 

L 



Page 41 



nrnora „ The longitudinal scudy of che Pasadena comp ed 

SllK No'lff b * lieve > do ^ as correccly as normal consCra incs 

t r f£ °? W u S made C ° otherwise ""it or define 

the daCa, Co make che program look beCCer. The sCudv included 

all students within che discricc who had entered thfschool 

in ?9?8 er f^H nerS , in , 1975 and Wh ° were sCilL 1" Che district 
in 1978 ac che end of grade' 3. Of che 824 sCudenCs ouc 
of over 1 800 who began in 1975, 640 were serve7?n our 

2 ™ P * d S fl ^H OLS, ^ iLe u 18 ? were served in our unfunded 
™ 5 fundamencal schools. AlmosC Cwo-Chirds of che 

disfrfct^i^T 3 3C ChS U funded Sites were in the 
aiscricc s chree lowesc socio-economic cacegories, while 

a^che'non 6 ^ ° f Ch * SCudentS "^ould 'have qualified 

In short Jr U , f S1CSS , WSre f ° Und in Che same three categories 
in shore, accual comp ed parcicipancs were, depending uoon 
your viewpoinc, half as rich or or cwice as poo? as fheir 
non-parcicipacing peers. v r 

Now for che resulCs. In reading che average como 
ed participant gained 12 percentile point! over the chree P 
school years while moving from the 28th percentile in the 
spring in 1975 to the 40th percentile in spring in 1978 
" h " founds unimpressive, you should remember that the 

hv tht Sf«?S r^- U u ed discounCs the inflated gains shown 
by the district which insists upon measuring gains from 
fall to spring. This growth is'real, and I bilievei? re- 
presents a considerable achievement in the view of the fre- 
quency which we hear and read about, the declines in student 
performance across the nation. scuaenc 

,een r-o k 0 " PaSaden f f chools >" to quoCe che sCudy, "can be 

IS5 rhS D f r success £ uLL y narrowing che gap between underachievers 

B t e national, buC rnychical, average scudenc, and even 
more so in language and maChemaCics chan in reading." For 

2i a 2iJ:«i?ii an8 "P t u he , avera § e com P ^ participant gained 
21 percentile points while moving from the 24ch Co Che 45ch 

?? r nf!! C i J-i In machem atics che average parcicipanc gained 
25 percentile points while moving from che 22nd Co che 47ch 
percentile. Given che naCure of kids, I find Chese *ains 
impressive. ° dUI!j 

Let: me illusCraCe chem in anoCher fashion. As 
schools including all sCudencs enrolled, whecher or noC 
served by comp ed , che eleven comp ed schools outgained 
the more affluent nonfunded schools in all three subiect 

n^nrc beCWeen , 1975 and L978 - Thg y did so °y 5 Percentile 
Points in reading, while moving as a grouo from che 43rd 

n^-nr u C t P^cenCile, an overall gain of 18 oercenCile 
poincs, while che nonfunded schools gained 13 percencile 
poincs. In language chey ouCgained Che nonfunded schools 
■4 percencile poincs, while moving from che 50ch Co che 62nd 
percencile. 

,>ed rocfi !J U ! ChS evidence .of success is based upon scandard- 
i^ed cescing programs, che disCricC also uses various cricerion 
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reference instruments , part icularly at the K-6 level, and 
has developed its own proficiency or competency test for 
use at the secondary level to determine graduation standards. 
We have used our criterion reference tests appropriately 
and extensively to insure that what is being taught is being 
learned, and to place students at their correct instructional 
level regardless of grade level. 

We have used our proficiency exams to insure that 
graduation from the Pasadena schools means something. We 
did it before the law required us to, and we are doing it 
with higher standards than typically are being applied. 
For example, when first tested in 1977-1978, 50 percent of 
our sophomores failed our proficiency tests. But, instead 
of lowering our standards for graduation, which would have 
been easier politically, we instituted remediation programs 
at the secondary level. In 1979, 61 percent of the students 
taking the exam for the first time passed. 

By the way, that is an interesting study which 
I would be willing to share with yea. It points out what 
happened to the students the second time, the third time, 
and the fourth time around the percentage continues to decrease. 
We are confident that these results will continue to shov; 
growth, that Pasadena's diplomas will continue to mean some- 
thing, and that eventually, before they leave high school 
all students can be proficient in basic skills, 

""he point of all this testing, however, is noc 
found in the results per se. Rather, their use in instruction 
and in communicating with the public is even of much greater 
importance. Each year ijvour district and our federal-state 
evaluation and test reports, we fully dissect the data for 
the purposes of improving instruction , for the purposes 
of revising and strengthening the curriculum, for the purposes 
of allocating funds and other resources. We do so publicly, 
by school, by grade, by program, although we have been extremely 
careful never to use our results for personnel evaluation. The 
fact that our results have been made public has proven to be 
quite an incentive for school improvement and for improving the 
reputation of the schools, sometimes if only for honesty within 
the community. 

Getting back to and on the urban intervention 
and evaluation, however, I am not sure what our experiences 
in Pasadena mean except that they are our experiences and, 
given the choice, if any intervention is to be done, I believe 
it ought to be done locally and evaluated locally rather 
than prescribed by the federal government or the state. 
While there are a wide variety of successful urban educational 
programs around the "country, they are successful as a function 
of Local involvement, development, implementation and evalua-" 
tion. I do not believe there are common elements or attributes; 
it they can be defined or divined, they can be formulated 
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Co provide more than a common sense understanding of. what 
ought to be done locally to meet the needs of kids, "of school, 
of their communities* I am not, however, pessimistic, except 
with regard to the ability of the state and the federal 
policy makers to impose yet another program upon all of 
us at the local level as a function of what you may-discover 
about some of us during a forum such as this. 

There are a number of lessons inherent in any 
successful program. The question is whether they can be 
mandated for application in other districts, in other circum- 
stances- Before concluding my remarks, I would like to 
discuss briefly what I believe to be the lessons of Pasadena- 

One, there is a wide variety of instructional 
approaches, of programmatic approches, which work and work 
well, depending upon management skills. Between the either/or 
situations presented to us by those who see the answer is 
ever more individualization and those who would have us 
see their answer in terms of a slogan," in terms of back 
to basics, there are an infinite variety of teaching methods 
and programs practices which work. Acceptance of this fact 
constitutes an acceptance of reality. Despite the crusades 
of the reformers and the mandates of reform programs, success 
remains a function of local application and not philosophy 
or legislation. ' 

Two, urban kids, any kids, do not need special 
education, with apologies to the advocates of 94-142. They 
need a good education. The difference between the two is 
more than semantic. Special programs, whatever their content, 
are too easily seen as appendages to regular instruction 
rather than as an integral part of it. Special programs 
are too easily seen as someone else's responsibility. Whatever 
the nature of the program's content, a school administrator 
must be made to be and feel as responsible for it as he 
or she is for the so called regular programs or curriculum. 

Three, everyone within a school, a district, a 
community, can or should, whenever possible, be seen in 
the broadest sense as an educator. This includes involvement 
of their parents, although too many educators confuse attendance 
at meetings at back to school nights, in resource rooms, 
with a kind of day to day involvement in their child's educa- 
tion, in their child's progress, that I feel is considerably 
more important. Further, when I say" everyone in the system, 
I mean just that. I mean that those procedures which are 
developed to comply with the dictates of the law or the need 
of management should be developed to support those services 
and staff members which are move closely involved with kids. 
Too often, requirements at the education center, and I refer to - 
my own office as "Puzzle Palace," at the district level are 
allowed to supercede those activities which are designed for 
kids. 
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Despite regulations and dictates of the administra 
tion, I have found that the administrators can Learn to 
provide services, supportive services and products rather 
than merely adding to the burdens already placed upon the 
rest of the system -. In Pasadena, such supportive services 
and products have led over the years to higher standards 
of program design and planning than typically are required, 
to on-site assistance for student identification, to useful 
testing and evaluation reports, to useful on-site review 
and assistance and much more in termS-'of curriculum planning 
staff development, implementation and evaluation • 

Four, none of these lessons applies unless they 
are accompanied by an honest and an open willingness to 
undertake the challenges of public education, of educating 
kids, a commitment to see the task through to its end each 
year with a new set of kids, with the same enthusiasm^ that 
kids as kids can bring to the process, and a belief that 
it can be done, that kids^can be' educated by us. There 
is neither secret nor panacea in which I believe ^to be Pas- 
adena's success, and there are many problems that remain. 
But I do believe we can have an effect on kids,* that we 
can educate kids for the better of us all. 

We can educate kids in the years and the- battles 
ahead. I would hope that the defense of public education 
can rest upon our successes, our products., rather than, as 
all too often has been the case, merely upon our hopes and 
our promises. There is much to be done and much to do withi 
the system and within the budget, without^succumbing to 
the illusions of reform and the enticement of new funding, 
or the temptation to so narrowly define proficiency that 
subsequent claims of successes belie our broader efforts, 
the expectations of the public, and the needs of kids. 

NANCY HICKS, 
PRESIDENT, 

THE INSTITUTE FOR JOURNALISM EDUCATION. 

I am a journalist, a sometimes educator. I have 
the feeling that with all the heat I had to take off my 
armor; it is armor that I put on when it was' explained to 
me that my role would be here today was v to explain how come 
the press kills everything before it gets started. 

MS. HICKS: I will" try to deal* with that. I first 

wanted to explain to you a little bit about how I come to 

these issues because it will affect what I have to say about 
them. . ' 

I am a product of early intervention in urban 
education in this form. In the sixth grade, in 1956, in 
New York City, my class at P.S. 46 was intervisited with 
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ch^ C lc laSS l ! dis ^ict. Afcer che Brown Decision, 
wnite kids. I don't know chat what we were supposed do 
Is that l lV ^r bUt 1 5 hink JPy most lasting ^ ° 0 f that 
and^hey'difnot?^" ^-cultural surplus in tL lunchroom 

I have been a working Journalist for 14 vears' 

? r V ' and Che New York Times," cutting 'mv teeth on 
he Ocean Hill-Brownsville school decentralization issSe 

infthft m s"ry? m:S ChM My J ournalisC "ving or dead ?over- * 

institute JiffL"?!!' I am - TtRfning a journalistic teaching 
institute called the Institute for Journalism -Education 
We have training and placement programs for minority ]Sur- 

p?o2?am't ha r 8 h^ S m We H T £ in ? ° f 3 Passional intervention 

?n ! = ? y ° f C r h , 6 im P e ^tives and problems of 

a covenan? W T' "V" nine j° u ™alists who took 
ten vea?s Af look ^ d u a ^nd our profession which, after 

D?ofeIsIonJr<: of ? f hard ?° rk ' SCUL has onl y 4 '95 percent 
professionals of color m the ranks of daily newspapers. 

K Qt ,,- n ^, u" 6 COuLd have Caken a structural approach or we cou]d 
begin etc hing away at che bl d t a little bit a- a 

Cirne. What we did was to set .up first a program to train ' 

?nd^trv V s ai 5 e K rt:erS C ° Place Chem in ne^paper jobs. The 

industry said there „as no one qualified'. So we said "We Wll 
give you someone," and we did that. Then it .ooked as if wp 

e^L'fre^t^slrbe S ° " ^ U g * ^™ ^""ejJeS- 
encea reporters to be managers and put them in newspaper iobs 

Well, we didn't need to train everybody. Everyone does not n L 

here Ca in°S- h S ° r & Set U? u * j ° b re£ ^ al ^rv^e tha? opiates 
halt iobs 8h S5 C on H so , ChaC Pf°Pl e who needed no training could 
finSJ- JS? ' ndus ^ r y couLd not say that there was no help in 
finding qualified professionals. • P 

r , , . Wh . at: v * did wich the industry was we gave them 

of w £ -f^ Th6y S3id We are noC qualified § the parallel 
of which is kids cannot learn. We said here, you have not 
welcomed minority professionals into this business That's 
the problem. It's your fault, it's not ours. What are 
you going to do about it? • 

The success rate has been quite something for 

rate^rhrhHc 3 ' We h V* ' ? e haVS abouC a 75 P^ent 'retention 
rh.r ha c An n6SS a f Cer . ch ree years, and this is a business 
that has a 40 percent attrition rate in two years. So we 
are doing twice as well as the average of black or white 
Reporters in the business. 

vo„ h™-,- When y° u look at intervention programs, I think 
you don t see that assumption in the school system, that 
the job is theirs to do and not the kids'. I was glad to 
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look at the literature for this program and see that that 
is turning around somewhat , The accountability of teachers 
and educators is becoming more the norm than not. 

But, for a large measure, a lot of the programs 
are oriented to behavior and not to learning, -While that 
may make for orderly classrooms and there n\ay be some better 
results on test scores, I can tell you what it means by 
the time students who have been in programs like this get 
to meet. What it means is that their drive and their spirit 
is taken out. We see more and more competent young folks 
who have no daring and no ambition and no ability to make 
the system work for them, I think as educators that we 
have to think as much about that as about the test scores. 
What are we doing in making nice, neat .programs and test 
scores if children are not allowed to disagree or to come 
up with their own formulations of doing things? 

I was very impressed in looking at the Larkin 
study out of Milwaukee for this reason. We have paid attention, 
right-thinking people on this issue, to che idea that every 
child is valuable. Well, every educator or every teacher* 
who is in position and who can't be moved aLso is valuable, 
I you have someone in the classroom that the- union says \ 
you cannot move, you'd better figure out how to make that 
person work for you. Hers was the first data I had seen 
on a program that went directly to trying to change the 
attitudes and performance of teachers who are in place and 
who you cannot move. You are not always going to be able 
to sidestep them. 

In our programs we are* very lucky as we can select 
. broadly and get who v^e want. You understand that every 
once in a while you are going to place a person in a job, 
for example, with an editor who is net very friendly to 
him or her, and you have to deal with that. 

All of this plays into the coverage a little bit. 
The reason we are in this business, what some people call 
social experimentation, is to change the portrayal of non-white 
people in the newspapers, v This is a sort of tail chasing the 
dog around. Part of the problem of expectation of educators 
comes out of what they read about minority people in' the 
newspapers or what they see on television. They seem to see that 
we are less intelligent, less hard-working, less able to 
concentrate. This is a function of having a cultrually biased 
press, 'which is predominantly white and middle class, in a 
country that is about 20 percent minority, ^as I think the latest 
census will show. They say that we don 1 1 know English, and this 
, has. kept us from getting jobs in tr.<* business and being able to 
portray a different life than that which you see every day, 

I would like to take a little look now at what 
the press, biased and unbiased, meets when it gets irto 
the educational coverage area, an admittedly flawed press. 
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I'm sorry Co say that I have found in the range of coverage 
that I have done in my professional career that educators 
are among the thinnest skinned folks that I ! ve run into. 
They are less straightforward, and I don't mean that they 
are not honest. But when someone asks you a question about 
how your kids are doing, the answer can't be, "Uhh," You f ve 
got to have an answer* If you want to know wha.t that looks 
like to -someone who is interviewing you, play it out on 
videotape once. Every pause looks like indecision and uncer- 
tainty and like you don't Tcnow what you are doing. 

And, there is a naivete among the ranks of school 
administrators, given the political clima,te in which they 
are dealing. Politicians think of the press as parents. 
We give birth to them. Educators think of us as assassins. 
Yet the public educators are as political and they should ' 
recognize themselves as such. In order to obtain the financial 
and other support needed to have a., intervention of any 
'kind, you already had to sell someone, and you may- or may 
not .know is going to exist. But the expectation is there 
for success. Part of it is your own problem. This, again, 
is why the Larkin program looked very attractive. They 
did not sell anyone on anything in terms of more money. 
They tried to reshuffle the deck and come up with something 
with existing resources. 

One suggestion I might make is to look at where 
you go for your experimentation money. Try to use your 
friends for that, your old friends. I know that we do that. 
When we want to do something that we are not sure is going 
to work, we go to people who know us well and who are willing 
to invest, to risk capital in our program development. 
We use new sources of support for things that are tried 
and true. r'~ > ^\ 

If you are going to try something that you don't 
know will work, if yea have an alternative, don't go into 
the public-political market place for money.' Try some other 
place. I think that might change the climate of expectation 
for your programs. 

I know that research is careful and that it takes 
time, but you have to have a way to have a short-term assessment 
of what it is that you are doing. I've read all of the' 
papers in the packet, and I've not seen one number. You 
need somethi' - concrete. The logicians tell us that things, 
that are concrete are proper names, addresses, and numbers. 
If you don't have a way of at least putting markers on some 
of the kids and knowing what you want to show, you cannot 
put off the public for that long. There are the politicians 
who haye the budgets to approve. There are also the parents 
who are offering their kids to your experiment. You have 
to be accountable iv the short range to those particular 
constituencies. You cannot be euphemistic. I've never 
seen a newspaper lead include the words "interface," or 
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"empirical" or "mention" or "fell." There are hard verbs. 
You have to think in terms of what it is that the public 
looks at. The soft pause, the "Hmmm, let me think about 
it, does not go over well in the public arena. It may 
be fine in an arena of peers, but it is not fine anywhere 
else. 

r 

I don't think that we are off the hook in regard 
to coverage, but let me try to put together a short list 
of things that you can do for yourselves and then bring 
to us. Number one I would think is to know what your are 
trying todo and how to measure it. Ask yourself the question 
at least in a way that sounds human and public. It may 
not be in the jargon in which you are used to working. 
But do think that there are other people who are going to 
have to support this. 

Pick your principals carefully, and, if you can't 
pick them, train them. If you can, have some contractual 
relationship with them over the results of your experiment. 
Do so if you can. I understand that this is a sensitive 
issue with unions and the like. But there might be a way if y< 
think about it, of putting people under a real or implied 
contract to the results that you want. 

Think of the children, of empowering children, 
not or their parents, because they ultimately can do it, 
even when we can't. I am a-parent. My poor kid shakes 
his head and suffers every day about what it is that we 
do to' his life. But you have to focus on them because they 
are the actors. The parents are not the actors in all of 
this. 

You have to figure out how to motivate your teachers 
or to get your teachers in a good economy. I think school 
systems will do very well now in finding talented people 
to teach school. But part of that reason is that the job 
market is so tight, there are people turning to teaching 
now who would not at other points. I think mv observation 
is that more teachers leave because thev do not have the 
support of the administration, rather than the difficulties 
that they have with any individual children. 

So structurally, in terms of your own resources 
and the human capital with which you are dealing, see if 
you can work in that area. Then, what do you do with this 
press that won't do what you want it to do? I .nust admit 
that you can't take my advice because for all of the programs 
that I've run, I can't get any press coverage either. Even 
though my husband is a newspaper editor , that does not help. 

Before the problem comes up, what you have to 
is find a source on your local newspaper who you know and 
to whom talk. Develop some sense of trust with that person. 
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I think that to feel that he or she knows that you are honest 
before the crunch comes so that there will be some expectation 
of how to deal when there is a real problem, • Be very clear 
. 4bout what it is that you are saying and try to summarize 
the main point of- what your are doing, 

, * 

When a story appears and you don't like it, look 
at the why of it and try to be somewhat detached. If it 
is a little problem, tell the reporter not to make a big 
stink about it, "I didn't like this, I thought you were 
unfair. Do it this way next time,' 1 If it's serious, then 
then you have a whole range of options. You can write a 
letter to the editor and it will be published. You can 
You can possibly write an op-ed piece for your own newspaper. 
Many newspapers more and more are offering their opinion 
pages to local folks to tell their side of things. If the 
paper has an ombudsman, and many are doing ^that now, you 
can appeal your case there as well. If it is a fatal problem, 
one that you feel has disrupted your ability to raise money 
or to nelp your children^, there is a court of last resort 
in th- newspaper industry now called the National News 
Council, It's located at Columbia University and it does 
put particular newspapers and television -stations on trial 
for misdeeds against the public, . 

KAREN LAMOUREAUX, 
LANGUAGE ARTS TEACHER 
PARK JUNIOR HIGH SCHOOL, CHATTANOOGA*, TENNESSEE,. 

This morning, when we got here and I listened* 
to^the rhetoric of the first few. speeches, I began to feel 
a little out of place and wonder whether I really belonged 
hg^e after all. But the more people talked', the more things 
came out to show me that we have a great deal in common. 

For instance, increased planning time was brought 
up by Dr, F&rrar when she was speaking of the evolutionary 
planning approach. We need this so badly. When we are 
dealing with federal programs, we are required to do more 
work, but nobody ever thinks about giving us more time to 
do it in. So I really appreciated hearing that. 

Technical assistance in introducing new programs 
locally is so important. Just because you are local does 
not make you an expert on organization, 

Dr, Hilliard's reiparks about cultural retardation 
in evaluators I appreciated greatly, I think that a teacher 
these days has to be a cultural expert. Anybody who deals 
with children in a public system needs to have a really 
honest and open approach to the cultural problems that we ' 
face, I f ve been thinking this for ten years and' I'm really 
glad to hear it brought out. 
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Dr. Gold's approach to realistic evaluation also 
is appreciated. He brought up the fact that if you have 
more than one program operating in a school, these programs 
should be tailored to meet each other's needs and to fit 
each other. Two programs that are opposing in the same 
school are horrible. It's just terrible to try to live 
through it. They play against each other. Teachers suffer, 
students suffer, and we all lose. This is very important," 

I was asked to sort of state some of our needs. 
Just briefly, we need program continuity. We don't need 
a program that comes in and gets the ax in two years. It takes 
five years, from what I've seen, for a program really to work. 
We need time to permit programs to evolve to meet the true needs 
of the community. When a program is interrupted to be appraised 
or evaluated continually, then the effectiveness of the program 
is destroyed. 

We need program synchronization. If we have, 
as we do, Title I, ESEA and PUSH all in the same school, 
it can be a problem unless the programs are tailored to 
fit, I can't tell you how important this is, ,When your 
objectives clash, then you really are hurting all of your 
programs and you are demoralizing your teaching staff in 
a way that you would not believe. Also, the children suffer 
as well. ' 

I have heard the terms "researcher" and "evaluator" 
repeatedly this morning, and a lot of you are professionals , 
in that area. I am not, by title, but I think I could be 
called a pure researcher because I deal with the real thing, 
right down on ground level. My real thing is right here 
with me, and I would like to bring her-^up at this time so 
that she can impress you the way she impresses me. 

YOLANDA BYARS, 
NINTH GRADE HONOR STUDENT 
PARK JUNIOR HIGH SCHOOL, CHATTANOOGA, TENNESSEE. 

I would like to tell you about PUSH/EXCEL in action. 
PUSH/EXCEL at Alton Park gives students a reason to attend 
school regularly. What I mean is that it encourages students 
like me to try to better our education. 

The program for pupils with perfect attendance 
increases t v he number of students in school. The Paper Action, 
the Coke PaVties, Hall Monitoring, the Governance Committee, 
and the Media Committee and Radio Goes to School are all 
part of PUSH/EXCEL. It is making students push to excel. 
It was founded by the Reverend Jesse L. Jacksbn to. promote 
excellence in education and was designed for all pupils 
regardless of race, religion and success in school. 
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1979-80 was the first year. The objectives were 
to improve self-discipline, study habits, and attendance, 
Artendance was improved by 1 percent, and the .activities 
and awards werp coke parties, trophies, certificates, movies, 
small dances, opeak Out for Excellence congest, the News 
Awareness contest, the Who's Who contest, and Radio Goes 
to School, 

The objectives for 1980-81 are to improve self-disc- 
ipline and parental involvement. Regarding parental involve- 
ment, parents are given certificates every time that they attend 
school activities. Awards will be made to the one with 
the most certificates at the end of the year. Parents volunteer 
to supervise lunch lines, games, hall monitors and so on. 
Parents are involved by Happy Grams, newsletters, monthly 
newspapers, the Bridge Newspaper and the Radio Goes to School, 

I'd like to tell you about Ra.dio Goes to' School, 
I am a former member of Radio Goes to School, The program 
is designed for students who are interested in radio. Students 
are auditioned by reading scripts. Ms, Karen Lamoureaux 
is in charge of it and she is also the faculty advisor of. 
the program. Radio Goes to School also motivates the students. 
It is another' reason for the increasing of the self discipline 
program. There are ten bright students on the progjtram, 
which takes place every six weeks because of the other schools 
that are involved. The program informs parents, gives students 
-the experience of working and speaking on radio and tells 
or talks about activities in and around school. 

During rcy three years at Alton Park, I have tried 
to live up to che standards that I have set for mv life. 
My achievements' include being on PUSH/EXCEL 1 s Governance 
Committee, It is a" responsibility of the Governance Committee 
to determine that our .objectives are identifying impediments 
to education. The Committee also is responsible for outlining 
activities intended to eliminate or reduce these impediments 
to education that I identified. Being a star-roll student 
in science, I maintained an "A" in the first and second 
semesters of eighth grade. Being an honor roll student in 
the seventh grade, I made the honor roll two times. In 
eighth grade I made it three times, and in ninth grade, 
I hope to make it all 'four times, I won an essay contest, 
coming in first place, on the subject of M Why I Like Chattanooga 
Public Schools," I am also the head majorette and the co-head 
cheerleader at Alton Park, To fulfill my needs of writing, I am 
writing for the "Bridge" newspaper,' Radio Goes to School is 
also my favorite activity. It gives me a great experience 
working withfradio. ' 

MS, LAMOUREAUX: I think you 1 !! agree that she's 
quite a little girl, just from what you've been able to 
see so far. 
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Whac I have Co say now is not so happy 'and noc 
so nice. The real reason that t 'brought: Yolanda here is 
as a graphic aid, a visual aid. I wane you Co see what 
an inner cicy sCudenC can be, can do. Then I wane you Co 
Chink abouc chis. Any number of her peers, kids who operaCe 
on her level, who produce wich Che kind of flair she produces 
with, who have Che kind of composure, creacivicy, and imagina- 
Cion chac she has, are jusc noc going Co make it. This 
SCudenC, hopefully, I pray, will make ic. My poinc is-ChaC 
Chings happen Co chese kids because of cheir environmenc 
which descroys Cheir chances. 

In Che audience righc now is an essav which I 
have passed around. Ic is wriccen by a young lady who is 
noc so forcunace as Yolands . This young ladv, vou will 
see from her essay, has creacivicy, imaginac'ion", inCeliigence * 
and she is Yolanda' s age. She is very aCCracCive. When 
she read char, in fronc of her English class, a number of 
people accually cried, ic was so good. She is pregnane. 
Her chances for success^ are practically nil. If ic happened 
Co her, it can happen Co any child in Che Inner ci.Cy communiCv. 
This is Che communicy chac chese ' programs are 'scruccured 
Co serve. 

So, when I asked for program concinuiCy, I meanC 
LC in several ways. FirsC of all, PUSH/EXCEL in our school 
and in our communicy is an efforC Co promoCe value. Ic 
is an efforc to ceach children che idea of excellenc, buC 
also Co give chem a sense of value abouC Chemselves and 
values, period. Now chey can come from home,'buC chere 
are so many chings in Che communicy chaC work Co destroy 
Chem chac chey need excra help Co keep chem up. PUSH brings 
in che churches. Where is a beCCer place Co go for value. 
IC brings parencs inCo che school. Ic brings up' che expecCa- 
Cions of che children chemselves. So, please, do noc cue 
ic off because Cheir social advances and achi <=>vemencs are 
in some ways much more imporCanC Chan academic achievemenc. 

The lit;cle girl who wroce che essay is gone. 
Her chances are over. You can whicewash ic all you wane, 
hue really chey are over. Her inCellecC has /loChing Co 
do wich ic. The communicy did ic Co her. Also, in chis 
area we have Co look aC che oCher pare of chis combinacion, 
and chac is che little boy who fathered the child. Generally, 
when this happens the child who fathers the baby is a non-ach- 
iever or a low achiever. !ie might be older, he might not. The 
fact is that he probably reads on a very low level. He either 
does. not have a job or can't get a job. He might have been'to 
jail. He has any number of problems. In other words, hi has no 
sense of value except as a father of a child. 

In this regard, we need to think serious Iv about 
continuance of Title I programs, programs that teach kids 
to read and which raise their reading level. This is most 
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important:. I can't Cell you how imporCanC it is for a young 

"EhJ* nee< ? ChiS Ver ^' ver y badl y' because thev cannot" get 

jobs. They can t compete unless they have this background. 

k *-c S , a LeCdown > 1 m sorry. You've just seen what 
a beautiful job Yolanda did. But if you don't see for real 
with your eyes and understand that it is not just the kid 
who reads on 1.0, it is a child like this whodso can b 
just chopped down. 

We need academic and we ne c . social suport. In 
the communities where these programs provide assistance 
iiKe this, help is going to come from no other quarter. 
.It. is up to you. I will close with this. I do not see 
^^ ng Wron f wiCh Poking to the Federal Government to 
provide social conscience. That is what government is for 

for Jh!i Way Si * ?" and , alwa y s wiL1 be-. I make no apologies 
for these kidl?* 6 ' CheSe P rograms going strong 

STUART RANKIN 

ASSISTANT SUPERINTENDENT FOR , PLANNING , AND EVALUATION , 

DETROIT PUBLIC SCHOOLS, DETROIT, MICHIGAN • 

. c , , c l want Co CaLk about evaluatiorTa's intervention 
instead of evaluation of intervention and wTll do so in 
= "Jinute. But first I had better tell you from whence I 



a 

come 



My present biases, and those of many of the intervene 

ou^f thp egLeS Vt g0ing ° n in DeCroiC P ublic schools are 

out of the research of Ron Edmonds, Larrv Lezotte, Wilbur * - 

?iSSe?r r 'n r nj ^ in B i°°!S' T ° rSCen Husen '< Rutter,'poDham, 
TidwelL and Courtney Casden and Lebov, a couple of psycho- 
Linguists, and Dave, who knows about parent behavior/and 

^? rh r , S r ein ', a - f0ll0Wer of u Pia S e ?> has extended some 

or that work into some other areas. 

r-hir^ -n If I . were c ° Ca]k °-^out what intervention programs I 
e ?lk "° rk p Ln , urban ™s and seem to be working, I would 

be that rLlal f0l i°^ n8 , characteristics. The first one would 
be that change probably happens school by school; than: is the 
single school is the largest unit of change and mavbe w u 
T^r^V^V^ We t re WOrking on intlrvention^ programs 8 

lorL oTi ^°rf ir V C I** SCh °° L 35 3 Social svsCem 5 t:he 
works of Lezotte, Brookover, Rutter, and Edmonds have done tha.. 

The second characte r i.st ic is that a good program is 
around rh^ VS OWnershi P; this means by all the key people in and 
JS?h?1 v £l S r ° gram ' and y ? U knOW who ch ey are. You have to work 
strate^ipf /S ^ !" VoLveiTient: sCr ategies, your communication 
IwLll 8 ' and u Y ou r implementation strategies to be sure that 
there is ownership by the people who have to make the program 

in ft PJ ob ? bl - v " ould be a better idea if thev were designed 

in the first place by the user, and then vou would not have to 
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worry so much about: whether chey can buy inco che program I 
can gee awfully exciced abouc scuff I designed. It if a litt'e 
tougher for me Co gee high on somebody elsf's design 

The Chird one is che one you have been hearin* 
c^ac k?d m ° rn , ing} WhiCh iS ChaC ce ^hers haJe Co be! eve 

a a^ r^ 1 Chln ! t al ° ng WiCh Chis 1 would «anc 
5p L y - Q , Can have an alc ernacive Co che normal curve. 

We don C have Co concinue ChaC forever. It's possible chat 
we could have equiCy in educaCion and chac we could have 
, f UCPUC , ter f' and Che distribution would look chT 

c seems Co'L 3 ^ f° 0r ' b i 3Ck and WhiCe = male and Ornate. 
It .seems Co me chac now chis is possible. I chink we know 

enough Co do chac. I don • C chink ChaC you can convince 

teachers Co believe ChaC kids can learn by Che Rosenchal- 

Jacobsoncon. I don - c Chink you can do it by Celling chem that 

crjey oughC Co read che research ChaC says ChaC c's 8 poss?bfe 

Edmonds" or anybody else's. I chink che only wav you are going 

t v L ° t ri C H m i° n t UCt: f hemselv ^ 'm ^e' classroom si 
rhinl I 8 6 feedback themselves ChaC chose kids can learn 

I think it is possible to surround them with the support 
scracegies chac will produce ChaC. If vou are inCeresced in 

ana" ee e if a fc? r ou e h C rit ^ be "« ^ be^vior change 

feedback tLr I t ChaC you can geC che kind of return and 
reedback that maybe will have an impacc on aCCiCude, buC ChaC 
will ac. lease reinforce che behavior change. 

, u Th r f u U ^, Ch Ching is chac teachers have Co believe chev 

than?? is cLc k"d, iC 13 , almOSC ^u rd6r C ° COnvince 'hei o? thai 

1 , C ? a 5 kLds can learn > although chese obviouslv are 

closely relaced. BuC it is a Cerribl v imporCanc th«n* to 

believe in if you are going Co teach. - ( Bv the way these icems 

are noc in any parCicular order.) ' V ' S 

a school rVS f ifCh thing - l S ^ haC none of Che scudies have found 
ins^uciion.f f successful where the principal is not che 

one lTt * ?hl i er ' , N ° W y °? may be able Co - 1 have not seer, 
one yec. The principal sees his or her primary cask as 

Progr^ 8 P erf ~ e ° f ^aff and leading che ins c rue Ciona 1 

earlier ^oh 1 ?"" charac CerisCic is one ChaC someone said 
earlier which I would echo, it concerns che wav che parenCs eec 
tZ lrlt s "?P° rC u ed ^aCion. This is in parenc behavior in 

L tH , gar in8 ,u- r J OWn e chlld ' raCher Cban as passive 
C t iS nor n - Ln . al J k t nd f of c ^y-wide commiccees and so on. 

" am noc againsc che 1 .ccer; I jusc don't see che evidence 
ot chac making a difference. nce 

Then chere are four chac gee more inCo classroom 

t?Z l ° r \ ThiS neXC iS Che hardest-one co sell, or 
. sound in Decroic Maybe it would be easier in other pLaces. 

aarn har-S ,? 6 ?nH- e -H d \ f ferenCiaCed instruction. We worked so 

darn hard ac individualizing instruction some 15 or 20 v»ars ago ) 
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that we got fairly good at it. We told teachers that if thev 
did a good job of teaching, chey would spread the distribution. 
Sure enough, in all those individualized programs of the 1960s, 
the rich got richer, the poor got poorer, and the mean stayed 
where it was. 

There is another thing that has come along more 
recently to reinforce that, engaged time on task, that vou read 
m the California Beginning Teacher Study. It seems to' show 
that you should choose to have the student actively thinking and 
engaged in the Learning task and tied into it. In other 
words, if you can either have ^adiness or you can have engaged 
time on task, ic looks to me as if the weight is on the engaged 
time on task. I don't think the story is aLl in on that. I 
would be interested in seeing some more research. 

This means that some of our grouping practices, 
not only chose of between class grouping but aLso the within 
class grouping, ought to be challenged . Another thing is that 
there is nothing so terribLy wrong about having a voungster 
practice the task in the classroom that vou reallv'are expecting 
him to Learn This means that vou had better define what that 
Looks like in a measurement way earLy-on. There has to 
be some feedback, too . 

Those are the kinds of things, it seems to me, 
as I Look at the research, that we now know. I think ten 
years ago, if we claimed we don't know how to teach urban 
kids, we might have had a better argument. But I chink now we 
certainLy have enough information sjd that the excuse ought not 
be there now. 

Now, what about Che Detroit program? Now I am working 
with Edmonds and Lezocte to try to put some of that research 
chat they have been doing recently into operation in Detroit 
schooLs. We have a Lot of activity going on there. That is 
s^hooL by school. What kind of support structure do you need? 

The program that I want to mention is one that 
James Popham^ and I put /together over the last three or four 
years. It ijs caLLed the Detroit High SchooL Proficiency 
Program. Ic, is that/ter ribLe thing of hanging a rap on the kids 
for the ilii of society, schooLs, and homes, bv having them have 
to pass an examina/ion in order to get a high schooL diploma. 
We are doing it. /We don't realLy withhoLd the dipLoma." We just 
endorse the dipLoma of those who do pass the test. 

We gaint-d the power commitment from the superintendent 
and the board eariy on. We have taken <:he troubLe to invoLve 
the community appropriately, and aLl staff and youngsters and 
enpLoyers to heLp determine whac the objectives shouid be. I 
think we have done a good job of defining the test specifica- 
tions carefully. An objective for us is'not a one-Liner. 
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Do you remember those behavioral objectives thpt 
we wrote in the 1960s and early 1970s? We finally ' broke 
down the curriculum so that it was so small that it was trivial, 
it was non-additive. It was multiplying a two place number 
by a three place number, and consonant blends, and things 
of that kind* 'So, haying done that, we ended up with 347 
objectives for an elementary reading program. I would defy you 
to find me a teacher who could teach 347 objectives in one term. 

We should move from norm referenced tests to criterion 
referenced tests because norm referenced tests are standardized 
by picking items that discriminate and items that discriminate 
are likely to discriminate because there are things you are 
not teaching or because they are biased in some way/ So, 
therefore, if you do a good job of standardizing the test you 
can come up with an instrument that measures what you are 
not teaching and which does so in a biased way. That is 
over-stating the case, of course. I think far more bias 
enters the testing situation by the person giving the test 
or before you get ready for the test. So we have to do 
a lot of work there, too. 

s In any event, it is clear to me that defining 

very carefully what it _is that you are expected to learn 
is important. In the first place, you pick something that 
is big enough, like being able to write a paragraph or being 
able to get the main idea out of a reading passage, or being 
able to use formulas in mathematics, something like that. You 
pick a chunk of curriculum which is large enough so that it 
is meaningful and small enough- so- that it is not still a 
goal, so that it really can be objective. Then you define 
the devil- out of it. This means test ✓ specif ications . It 
means very specific statements about what the stimulus in 
those te6t questions will be, what criteria the resp ->ses 
have to meet, what the suhskills are, what some teac g 
strategies are, and then you lay all of that out so that 
you' have a full instructional program. 

Now you have done Che job of analyzing what it means 
to learn how to write a paragraph. In Detroit, our high school 
English teachers are now telling us that within the last year 
they ^finally have learned what it means to teach paragraph 
writing. They have learned it because we did a pretty good joti 
of defining what that looks like, we are clearly measuring it, 
and the chips are down. Incidentally, it does take a little 
time to learn how to score 30,000 "br 40,000 paragraphs and 
to do it in a fashion that is fairly reliable. We have learned 
a bit about that. 

Good learning is happening because the attention to 
appropriate measurement was given, because the objectives are 
few in number and very well defined and very important, because 
there is involvement and commitment and ownership, because the 
stakes are high, because there is a full support program of 
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in-service education, materials, and all of that to go with it, 
because there are practice tests, because it has been fed into 
the curriculum K-12 not only in English and math, but in other 
areas, because there is a remedial program in secondary schools, 
because we are using what we know about peer grcup tutoring and 
.cross-age tutoring, and because we are supporting the program. * 

I think it will work.,. I don't know yet. We -gave the 
test last January'for the first time to our tenth and eleventh 
graders. 82 percent of the youngsters passed the reading 
portion at that poinr; 55 percent passed the writing portion and 
50 percent passed the math portion. As you know, most of us 
have been emmphasizing reading more than we have those other two 
areas in our schools in recent year's. 

If we are talking about evaluat ion .of intervention 
programs, or even evaluation as; an intervention program, 
then we need to look at the beginning of the,, evaluat ion 
process and the end of the evaluation process more carefully 
than we used to. The middle is the data collection^ the data 
organization and the data analysis and so on. The beginning 
is the focusing, t the designing, the who T s going to make 
what decisions on the basis of this, when are thev going 
to make them, what are -£fcte objectives, what Hoes this reallv 
mean operationally and so^so. The end is that the evaluator 
does a decent job of not just handing somebody the report, 
but seeing that the report is interpreted, understood, and used. 

I think we need to work at those two ends as evaluators 
if we are going to hel-p improve urban education. 

ROBERT STAKE, 
- DI RECTOR , THE CENTER FOR INSTRUCTIONAL RESEARCH 
AtfD CURRICULUM EVALUATION, 
UNIVERSITY OF ILLINOIS, COLLEGE OF EDUCATION. 

I will try to "limit myself to three comments. 
We have been on quite a tour of" urban education evaluation 
problems this morning and the variety is almost overwhelming. 
I will try to draw attention back to' three things that our 
first three speakers said. 

First was Norm Gold. How refreshing, to me, anyway, 
to hear an appeal to conceptualizing the program evaluation 
responsibility around the expectation of stakeholders. 
I say this not because I aa^ but that is not why. It is 
It is because that is the richest source of ideans for under- 
standing what is going on in that program. I Relieve that 
Norm did />ay that the evaluations i-hcuid be concerned about 
modifying the expectations, making them more realistic. 
I wiLl argue with him on that one*. I think we are too inter- 
ventionist already in our evaluations and it is not our 
role, to make expectations more realistic. 
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Next: I will comment: on Asa HilliarcPs presentation. 
How delighted I was by much- Chat: h*e said about the use of 
observations, about the cultural deprivation of evaluators,' 
and about content validity , I was distressed by his optimism 
that scholarly or measurement technology has in the foreseeable 
future too much of anything to contribute to program evaluation 
and by his failure .o doubt more generally the common concept 
that aggregating information 'about individuals, their gr.owth, 
their education, tells us much abut whether or not the program 
is operating in a responsible way according to ordinary 
expectations of what teachers and other people believe that 
should be doing. 

The principal eye, it seems^to me, of program 
evaluation should be on the professional educators, not 
On the students, asking partly whether those professional 
educators are' attentive of those individual children. 

-Next I would comment on Eleanor Farrar's remarks. 
How delicious I thought it was to lead us up to the doors, 
telling us there is a tiger behind one and some more clues 
as to how to choose the doors. The implication was, of 
course, that if we in program evaluation see the evolving 
program as an evolving program, much more than as something 
that is fitted out at the factory, in Detroit or elsewhere, 
we will attend to the important issues, considerations, 
valuings and language — and much of it will be street language 
-y- that is there, and that this is our job, I see this 
to be the case and there are various types of case studies, 
ethnographic studies, and other studies, as has been implied 
eLsewhere this morning, that are most useful- in making this 
record. 

There still is a grand expectation that I tfon't 
know what we can deal with,' It was raised in the .discussion 
first off this morning. There is an expectation that. the 
program evaluation is going to uncover some^f secret of innovative 
education, some secret of :eaching, some understanding about 
the processes that has not been recognized before, the aim 
of most research studies. 

Program evaluators are not going to discover anything 
more with the time and budget constraints ^they have,' Perhaps 
even if they did not have those constraints, they are not 
going tu discover more than is common knowledge to different 
persons involved with that program. The -secrets have to be 
looked for. We have to try to understand teaching and all that. 

But that is not going to happen within program evaluation, and 
it should not be organized around discovering those secrets. 

What we can as program evaluators do is capture 
some^ of the experiences, the insights, the judgments of 
many people, right there at the site, and of some who are 
looking on more distantly, and make a sensible composite 
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of these with some observations and scores of our own and 
we will use extracts from documents we can pick up to tell 
a story of that particular program, its context', its problems, 
its pride and joy, and to leave largely for others to make 
the more final decisions as to the success and failure and 
desirability of use elsewhere of that program. 

DISCUSSION 

GWENDOLYN AUSTIN, Teacher Corps: I wanted to * 
direct my comments to Ms. Lamoureaux. I think you have 
done an excellent job, I think that you are an asset to 
the school. But there is one request that I would like 
to make. I really became concerned when you made the statement: 
about the pregnant student thSt her opportunity is over 
and that there is no hope. That hurt me because if this 
v kid in the eight grade is lost, then there are a whole lot 
of them who. are lost throughout the nation. There are a 
whole lot like her. These kids, with the encouragement 
of their parents, many of them have been able to pick themselves 
up and go on to greater things. 

So, if PUSH is doing such a good job with parental 
involvement, then it needs to extend itself outside the 
school, to get to that parent ^nd encourage that parent 
and apprise that parent of the opportunities that are out 
there to help that kid, raise her hopes and expectations, 
and development potential. 

KAREN LAMOUREAUX: Thank you for your compliments. 
This is why I said that I hope sincerely that PUSH will 
be allowed to continue because somewhere along the line this 
little girl's failure was due to the lack of support at 
home. Somewhere along the line maybe PUSH can get to the 
parent. 

I didn't mean that a pregnancy causes all students 
to fail. I was thinking really particularly of this one 
LitcLe girl. Given her character, the lack' of support at 
hole, and the pregnancy, her individual chances for succeeding 
are not assured. So, please, don't think that I believe 
that anybody must fail. But her case really got to me and 
I guess I wa^ a little emotional about it. 

YOLANDA BYARS: I want to say that PUSH/EXCEL has 
gotten into contact with Tammy's mother and it still hasn't 
done what it should have done. PUSH/EXCEL is a great program 
at Alton Park, but to a certain extent, as in Tammy's case, - 
it was nothing. 

JIM STEPHENSON, Teacher Corps: On your concerns 
about the role of the stakeholder and given the fact that 
there is, no indication that this agency, the federal level, 
has in the past or would see nothing pending for the future. 
Would you comment, Dr. Stake on that problem, versus the 
continuation of a national large-scale impact evaluation? 
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DR. STAKE: In the past, there has been a great 
emphasis on the conceptualizations, the terms, the criteria 
of sucess and the like in terms 'of program designers and 
developers. Some people say well, those are important stake-, 
holders, and, fo be sure, they are. Other stakeholders 
at, the scene, those people who are going to consider the 
problem „as possibly useful in their school next year, the 
parents of the present children or the parents of children 
in the future, those stakeholders largely have been unattended 
to in the program evaluation efforts. I believe it was 
Norm's contention, with which I certainly agree, that, although 
it complicates things terribly, it -is a reality with which 
we have failed to deal and with which we should be dealing 
more, the multiple expectations of different stakeholder 
groups . 

SHIRLEY JACKSON, Director, Basic Skills Program: 
I guess I hear a bit of confusion, and I think this is the 
state of the art, in terms of doing naturalistic research, 
ethnographic research, and the preference there and the 
need for identifying student achievement gains. I am hearing 
a dichotomy there. We are saying both. In the school effec- 
tiveness studies, one of the things-that you get all the 
time is the assessment aad the monitoring of student progress,. 
So, somehow or ether we are going to have, to intellectually put 
those two pieces together, and I'm not sure that we have yet. 

DR. COMER: The monitoring of school achievement 
and performance I think is important and determining the 
criteria of what determines success is important. What\ 
I keep worrying about and what I did not hear even todav 
is the question of what is important in moving from all! 
of che things that we know to be involved in successful 
programs. What is required to remove unsuccessful programs 
from where they are to where they need to be? That, it 
seems to me, is something to which we are 'not paying enough 
attention. It is involved in the dichotomy about which 
you are concerned. 

It seems to me, that we have observed that are 
necessary and critical and must be involved and which we 
must pay attention to if we are going to learn how to move 
unsuccessful programs to successful programs. There are 
specific interventions that are important in doing that. 

DR. STAKE: I was^oiore opt im is tic four years ago 
that we could work out technical procedures whereby the more 
scalar psychometric reading score records could be combined 
with observations and that we could validate one against 
the other. I am much less optimistic now. We have tried 
to do it. We may not be smart enough in my group. Others 
should keep trying. But we are not' able to put them together 
very well at the present time. 
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My feeling is that it is better at the moment 
to lean on something that I believe is deliverable, that 
is, good representations of the process with the attendant 
information about individual case, classroom, school achieve- 
ment; but that you are not going to be effectively able 
to aggregate that across schools, cities, and the country 
to get a picture of success of any urban program or any 
other educational program. The aggregation skills are not with 
us at the present time. 

Both of the curiosities for the process and the 
impact are legitimate and should be pursued. Our abilicy 
to deliver at the present time I believe is much more in 
the area of describing what is going on in the teaching 
and activity of the classroom and the school and much less 
in the area of the reading scores. 
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